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!• EXECUTIVE SUMT^ARY 



This Final Report, and the accompanying documents listed 
on page 8 9 describe the physics course delivered under con- 
tract Number N00600-6 8C--0749 , its development, validation, 
and installation at the U. S. Naval Academy. The materials 
to be used in the course are separately packaged and are 
described in the letter of transmittal. 

- The course, as delivered, is self-paced, independent 
study, multimedia, computer or manually managed, classical 
introductory physics. It is completely packaged and can be . 
used at the U. S. Naval Acader^y with any number of midshipmen 
in the fleet, or, at any other place having a need for the 
content contained in the objectives as listed in "."^echnical 
Reports 5.1, 5.2, and 5.3, provided that adequate faculty 
support is available at student request, 

A second major component of the delivery is the Empirical 
Course Development Model v/hich sets forth the procedures for 
developing new courses or adding content to existing courses- 
Capable professionals, through the use of the model, can 
design and develop self-optimizing courses or segments of 
courses. 

A third major component o5 the delivery is the report on 
the research, evaluation, and validation procedures followed 
in" bringing the course to completion. This information is 
contained principally in Technical Report 5.0: Final. Report . 

The course is fully operational at the present time, 
the Physics Faculty can decide, at any time, to offer the 
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independent study course ^ or to continue the present procedure 
of offering the students a choice between the new course anc 
the traditional course. It is the contractor's recommendation 
that this decision be reached by the Naval Academy^ taking all 
relevant factors of current operations into consideration. 

The benefits to be gained by offering the independent 
study version should be in the ability to schedule the course 
on a flexible basis at all hours during the day thus avoiding 
conflicts with other courses. Further^ through time, a 
gradual reduction in the nvimber of people required to teach 
the course should be reaiized^ if this reduction should be 
desirable to the Academy. 

Because of the backgrcur.f and capabilities of the mid- 
shipmen, we have concentrated cn print r:.edia and de-emphasized 
the more complex and expensive ^^zic'Ls^al approach. The 
Empirical Course Development Mcfel does, however , outline a 
procedure for developing, usir.g , and evaluating available 
■audiovisual media. The basis for the decision is presented in 
this report, particularly in the results and discussion, and 
in the Revision Process Documentation. 

The fact that the systems approach to course development 
has been effectively employed in this course should be care- 
fully considered in analyzing future needs. The contractor 
believes that an extension of the systems approach to other 
courses and programs at the Academy would have benefits well 
above those possible to achieve in a physics course alone. 
Particularly, in the Effectiveness Rej^ort, the point has been 
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made that an absolute standard does not exist for the amount 
of physics that should be learned in this course, and there is 
no specification of the level of mastery expected. Further 
study of this question should yield a definition of physics 
knowledge requirements for students with any Academy major. 

Finally, the installation and operation of the course at 
the Academy during 1970-71 should represent only the beginning 
of the optimizing process. The course is designed to iterate 
and to be systematically revised. In this feature lies the 
power of the development raodel. 

The course is completed. It is quite effective. It is 
installed and operating at the Academy. Additions are being 
made to it according to the procedures recommended, and it 
should be revised and improved following each iteration. 
Management attention should he directed tov/ard seeing that 
the continuous improvements are made, that course operations 
are adequately supported with staff or computer assistance / and 
that .staff levels for course operations are consistent with 
overall needs of the Physic Department and the Naval Academy. 
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II. INTRODUCTION 

The Purpose of the Contract 

Exerpting from the original Request for Proposal (RFP) , 
the objectives of the contract were, "...to develop, test, and 
evaluate the best possible instructional media, materials, and 
strategies, utilizing all available techniques in the current 
state of the art." Within the context of this objective and 
listed later in the RFP are requirements for tryouts, re- 
visions, and evaluations of the various media and materials 
with the intent of including those naterials which tend to 
optimize the course results and eliminating those materials 
which do not appear to contribute to student performance. 

In any research and develcpnental effort of the magnitude 
of this contract requiring state-of-the'-art technology there 
are generally three sources of infonr.ation which emerge as 
relevant, and from which one can infer the ultimate intent of 
the contract. The first source includes the RFP and initial 
contract documents. The second source is the modified contract 
and official agreements that have been worked out between the 
contractor .and the customer. The third represents the forraal 
and tacit agreements reached between contractor and customer 
staffs as they progressed toward the delivery of the final 
system. These sources of information will be discussed in 
order and hopefully will provide the basis for establishing 
clearly the intent of the contract as interpreted both by the 
contractor and the customer. 
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Throughout, the RFP, the requirement for design, development, 
try out, and revision of instructional materials is reiterated. 
In addition to the instructional materials, various approaches 
to the scheduling and operation of the course had to be worked 
out so that the finally delivered package included not only a 
. set of effective"'" materials but also a set of procedures 
which will allow the Academy, in future iterations of the 
course, by evaluating and revising, ultimately to achieve an 
optimum course based on the original version and techniques 
applied by the contractor. The Empirical Course Development 
Model (TR 5.7) sets forth the rsccrjr.ended procedures for 
future revisions. 

Thus, two important considerations are established, (1) 
that the course must be develG;r-3f acccrzing to an empirical 
methodology, and, (2) that the fi-al ir:zleiuentation package 
should include techniques for rei-eratiz-n of the course with 
successive improvements to be riade by the faculty in the future. 

Since the term, "best possible'' is not officially defined 
in the contract or the RFP it was assumed that the meaning of 
this concept should be derived fron the contractor ' s analysis 
and approach to the problem so that the best possible finally 
delivered course would include not only the instructional 
media, materials, and strategies, but would also take into 
consideration the realistic constraints of the Naval Academy. 

1 . 

See Technical Report 5.6: Effectiveness Report for a 
thorough 'definition and discussion of "effectiveness." 
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Such constraints, likely to be found in any university, include 
the judgment of faculty meinbers based on their knowledge of 
content of other courses, the extra-curricular activities at 
the Academy, and the long range goals and objectives of the 
Ac ademy . 

During the materials development effort for the initial 
tryout, a number of discoveries was made by the contractor which 
resulted in design changes and allocations of time and re- 
sources for the computer. While it was initially considered 
to make the computer the central feature of the entire course, 
the results of outside investigations and recent experience 
at the Academy suggested that a redefinition of the computer's 
rple was imperative if the course vere going to be used 
successfully. It was agreed th^t the cc-puter was best used 
as a management tool (Computer M=r-=ged Instruction) rather 
than an instructional device, e^ir-srr frr a specific application 
of computers in solving physirs rrrbler^ vith many variables. 

The third source of speci f i tlons and agreements on the 

■.final package emerged as contractor and Academy staffs co- 

j 

operated to implement the interiia versions. It was during 
these tryouts that the various possible roles of instructors, 
amount of dependence on the computer, the function of the 
laboratory, and the overall operational methodology of the 
course were worked out to fit Academy requirements. 

Several additional reports have been prepared to be 
submitted in conjunction with and as a part of this Final 
Report which speak to various issues and aspects of the course 
as developed and finally submitted. 
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Documentation Reports 

1. PINAL REPORT (TECHNICAL REPORT 5.0) 

A. Description of the methods, activities, materials 
and reports developed and produced by the contractor to 
satisfy the requirements of contract N00600-68C-0749 , and of 
the Request for Proposal as a part thereof. 

B. Statistical data and statistical tests where 
appropriate , and , the conclusions drawn and recommendations 
made by NYIT on the basis of the analysis of these data and 
of the experience gained during the project. 

C. Specific reference to the Evaluation and Validation 
Design (Technical Report 4.7) and an interpretation of the 
data collected as specified by the Evaluation and Validation 
Design. 

D. Based on the foregoing empirical data and exper- 
ience, revisions to the Design fox the Selection of 
Strategies and Media (Technical Report 4.9). These recommen- 
dations and revisions are contained in Technical Reports 5.1, 
5.2*1, 5.2.2, 5.3, 5.4,. and 5.5. 

2. COURSE DESCRIPTION (TECHNICAL REPORT 5.1) 

The course as delivered can be used effectively as an in- 
dependent study, self-paced computer managed introductory 
physics course* The course materials can be used as supple- 
merits to traditional instructional techniques, or through 
appropriate management and staff assignment procedures, the 
course can be used to increase the number of students taught 
by each qualified instructor. Particular configuration used 
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should depend upon the needs of the Naval Academy at any given 
point in time. 

3. COURSE objectives' (TECHNICAL REPORT 5.2.1) 

Each of the performance objectives is represented by a 
problem so that the level, scope, and assessment measures are 
described in unambiguous form. The principal requirement for 
student success is that he be able to vzork the problems 
appropriately in the time allowed. 

4. COURSE STRUCTURE AND SEQUENCE (TECHNICAL REPORT 5.5.5) 
The topical sequence of objectives including the decision 

processes which led to this sequerxce. 

5. TEST ITEM BANK (TECK^ICAL ?Z?0?.T 5.3) 

A compilation of critericr. ch-~ck iter.s and diagnostic test 
items identified by terminal objectives. The item bank in- 
cludes multiple questions for ez^rr. terrr.lr.al objective and item 
statistics collected during the -ry:::ut conducted in the Fall 
of 1969. 

6. 1 MANAGEMENT SYSTEM REPORT (TZCHNICAL REPORT 5.4) 
A description of course implementation procedures re- 
commended by the contractor, the nature and form of the test/ 
the method of scoring and recording scores, the kinds of feed-- 
back provided the students, and the method of presenting the 
feedback. A description of all record-keeping procedures and 
the forms on V7hich records can -be kept. The report details the 
use of the computer managed instruction system where applicable 
7 . REVISION PROCESS DOCUMENTATION (TECHNICAL REPORT 5.5) 
A description of the specific empirical revision activities 
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rationale for these activities, and a compilation of the data 
upon which revision decision were made, 

8. EFFECTIVENESS REPORT (TECHNICAL REPORT 5.6) 

The effectiveness report discusses effectiveness first 
and primarily in terms of "mission" effectiveness, by ex- 
plaining the procedure used to derive the mission for the 
course, by elaborating a definition of effectiveness in terms 
of this derived mission, and an explanation of how the learn- 
ing system developed in this contract achieves the derived 
mission, 

9. EMPIRICAL COURSE DEVEL0?!1S1;T MODEL (TECHNICAL REPORT 
5.7) 

This report presents the final version of the Empirical 
Course Development Model, expanded to include additional tech- 
niques so that the model will r-r Errlicable to a .broad range 
of courses in instructional settings. 

The Empirical Course Develcrnent S!onel is presented as a 
complete set of procedural guides / with supplementary devisions 
and explanations. The model was designed^ first, to provide 
the Naval Academy Physics Department with the tools and pro- 
cedures necessary for continued course optimization, and, 
second, to furnish these same tools to the Academy and other 
schools and colleges for empirical course development in any 
subject matter area* 

The Empirical Course Development Model is intended to be 
used in conjunction with the Management System Report, Tech- 
nical Report 5.4, and the Effectiveness Report, Technical 
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Report 5 • 6 

These repo^ ively, are submitted with the in- 

tention of an, i least these funamental q^ostions: 

1. How did the contractor fulfill the terms and condi- 
tions of the contract as amended? 

2, What materials and level of detail are necessary in 
order for the Physics Department to understand the development 
process as it was practiced by the contractor, and, what 
specifications procedures and plans are necessary to allow the 
Physics Department to continue course development as re- 
commended by the contractor? 

We believe that the reports as submitted are sufficient to 
achieve both purposes, without containing unnecessary detail 
and recommendations beyond the scope or training of the Physics 
Department/ 
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III. PROJECT DESCRIPTION 

Overview 

There were three imp .ant milestones in the development 
of this project. The firni was the development of the content 
and objectives of the entire course, the second was the major 

. tryout in the Fall of 1969 of all of the learning materials 
at the Academy, and the third was the implementation of the 
course as it was intended to operate in the Spring and Fall 
semesters of 1970. The initial tryouts of the materials at 
the Academy occurred during the last part of the Fall of 196 8- 

. and during the Spring of 1969. The Fall 196 8 materials were 
the first rough draft version and were used with only a few 
students in order to determine level of expectation, quality 
of materials and time requirements. The second tryout in- 
volved a considerably greater amount of material SEtd laat:e-d 
for the aintire Spring 1969 semester with approxjTTTrrtely one 
hundiasd atodents . On the basis of these two eariy iterations 
of the imEttsrorials the configuration of the course in the Fall 
of 19 69 «i^::!idesigned, 

Thetprtrpose of the Fall 1969 tryput was to compare students 
preference £or, and, the performance of the instructional 
materials selected and packaged for the course. On the basis 
of perfaonance and student preference data, combined with 
judgments about costs and general effectiveness, the com- 
bination package of the final version of the course was to be 
derived. After the Fall 1969 tryouts, changes took place 
in the method of course operation, and the course content. 
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Further, Academy faculty had the first opportunity to develop 
materials according to the Empirical Development Model » The 
faculty materials development effort was concerned principally 
with addin^^ content to the original course in order to meet 
newly • resc ' ^d Academy curriculum requirements. 

Extensive data was collected during each iteration of the 
course for the purposes of making judgments about materials 
adequacy and for providing information upon which revision 
decisions could be based. Several large boxes of computer 
printouts, including individual student response records, 
synopses of group response records, performance data, rating: 
data, and time data have been collected and are available for 
deferailed review. It should be err.phasized that these materials 

not useful fsasr: summative evalisaticn. The iterations 
duyfirng the Fall J^cr; 19 69 and the firing of 1970 provide data 
usEful to ascert^EEt total value J0^±he course. Specific con- 
cih3EB:ions and recaminendati'ons aboirt the course and its oper- 
atTTons are made in Empirical Course Development Model (TR 5.7) 
annr:.in the Effectiveness Report (TR 5,6). 

The data presented in this report were collected during the 

19 69 tryout, 

3!feTall, 1969 Tryout 

The project design led up to the Fall 1969 tryout as the 
ma^or data source for materials revision purposes and to the 
Spring 1970 tryout for the final data source for management 
derisions. In tiie Fall of 1969, midshipmen were required to 
ccnnplei:e the ent±re sequence of instruction, * regardless of 
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whether they were capable of ccanpleting the course early. 
This procedure was followed in order to retain the complete 
Academy range of talent in the samples. During the Spring of 
19 70, midshipmen were allowed to complete the course early, 
r>- ''.xrjf^^ that they passed stringent criteria, in which case 
they were exempted from the final examination. 

The major Technical Reports,- submitted in 1969, describe 
in detail the objective considerations and rationale for the 
Fall 19 69 tryout. These reports are: 

1. Technical Report 4.7: Rationale for Sequencing 
Objectives- Finkel, 19 69. 

2. Technical Report 4.7.1:. Evaluation arid Validation 
Design. DBi^esrMssm^ and Branscn ^ 

3. Tecfmaksl Report 4 ,7.. 2 The Validation Process. 
iBteterline and Brannson, 1969. 

4. Technircal Report 4.9: Zesirr, for Selection .o£ 
Strategies and J?aed±a. Deter line =:r.£ Branson, 1969. 

5. Technical Report 4.3: Course Revision and Re- 

• I 

structure. Vieriing, 1969. 

6. Techiaiical RejHDxt 4.12i Weekly Course Segmen± 
DocumentaioDn, Ifeeks A through O. 



14 



Hiescription of the Course 

The version of the physics course delivered to the 
^ademy includes instructional materials, a management system, 
^d, recommended evaluation procediires . The instructional 
^materials were packaged to be used within the recommended 
oEanagement system according to the procedures described in 
ZSEechnical Reports 5.1, 5.4., and 5. ,7. The configuration of the 
-caourse is independent study with faculty management and support 
:SD.r personal consultation upon reqissst by the student. The 
-3EStructionaL aaeterials consist of tiie basic textbooks, the 
j aa'Mb 1 «n m s and solution books, —he cxiS^rion test items, and the- 
seJisaittsd audiovisual materi=±s. 

recommended confij5:;rzs:rlcnir^-^r^ cjDurse and the basis 
Soar it, are completely, respc-sdve to: tii& RFP , particularly in 
tfearjfis Q'f tha explicit objeciJ— es set foj=±i on page one: "The 
idfeseasxaiTzies of the program ax.e to de;^x)p -test and evaluate 
-iiaBibe^ possible instruct i r r- ^^ neSs-, :materials, and strate- 
0m&^ H±±lizing all a.vaila£23s: ::fcechiriq.tss ijl the cur sent state- 
rrf-^fiErrart. " ElsewhCT-e in -tfee- RFP speciSc reference is made 
im' Ca!B5).uter Assisted jEnstruc±xon (CAI) and Computer Managed 
^isstxmdrron (CMI) and±±he utilization of appropriate in- 
^ ifcUCfe^aial media., pajgticulariy audiosdsual materials. 

(jat ±iie b.asis of the data avaiiabl-e when the RFP was issued, 
r^Mfe were excellent jceasons to beTr-eve that specific audiovisual 
: tifflafiaaiis., Coi^uter Assisted Instruniion , and Computer Managed 
-Mstjrmsi-mon would all contribute un±q.uely to the performance 
tsssr iniifeiiipmen in the physics course. Our general results, as 
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well as results reported by other investigators, seem to in- 
dicate that the differential effects of media in experiments- 
on highly verbal students are rarely convincingly demonstrated. 
The contractor, therefore, has taken the view thar nothing 
should be included in the course which added either to student 
time or to course expense unless a clear demonstration of Lts 
utility could be made. The statistical analyses, results, 
and discussion sections present the data and conclasixEsas 
reached by the contractor after careful evaluation of "jzhe 
media and course structure. 

The report of an exhaustive search of the literatrcre by 
Dubin and Taveggia (1968) coirSrm the difficulty of 
demonstrating statistically ::s^nificant differences in "teaciiing 
methods at the college level- Indeed, if it were not for ±he 
finding that time spent in stes^' did contribute to improved 
grades for college students , ^feMsix analysis would have yielded 
no positive results at all. ]SE^t:r findings do not mean that 
students do not learn, but, s^s^y that they learn about as 
well uncJer virtually any teax22Efe:.:.g method •* • 
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IV. STATISTICAL ANALYSES 

Introduction 

In this section analyses are described testing population 
hypotheses using samples of scores from weekly posttests (both 
total correct and log confidence-^ scores) , the final examination 
and reported proctxDr time. It is assumed throughout, with the 
usual misgivings'^ rthat the underlying assumptions are valid 
fox: condaaxrting tiaa. statistical tjests and procedures . In those 
p3t»cedur-eE: where -aae calculatiorL of the power of the test is 
aiJimropri a±g. t±ie: jSaxmula 

h^aE;-:been uised to calculate the power. 

PcJWffiar-rhera,, as in all statisiiical discussions , refers to 
the^ipro brfeul itg?^ :o:f: rejecting theimull hypothesis when in fact 
iti^Ls f aT?Tt? , :I±:j±s^ desirable thant power be as close to 1 as 
po^:ibl&: sEEd t±i±s ^generally regaifcres tfeat the sample size in 
the . anai^is bass qpiite large. SiiJEce the samples used were 
laosge the obta±n:ed power value wsre abxa^^e .99 for d = .5a 
Cohen (1969) sji^gests -that a powex .80-,is sufficient for. most 
behavioral stiid-Les. The symbol "S" is :.;effect size and re- 
flex:ts how muck difference one is willing to tolenate before 
declaring a sigiiiJEi.cant: difference exis±s. If one desires 
to detect very small differences .-then d is small and large 
samples are generally necessary. The reverse is true for 
detecting large differences, i.e...., d large. Throughoxit this 

The log conSdenoe scoring pxcc^eatere is detsSled in 
Ifecdsmical Re^rt 4.7: Evalnmiid^Sjto: and Valid^iSion DesigH^^, 
page 28ff. 
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paper d was set at .5a, Cohen (1969) calls this a "medium" 
level. The symbols and frg denote the standard normal de- 
viate values (two-tailed) for ci ^nd g valuer respectively. 
Unless otherwise noted, * and ** will respectively denote 
significance at the .05 and .01 levels with cx set at .01 or 
smaller* 

It;canm3d: be over emphasized that the reader must reach 
his own. :concliis ions as a result of the statistical tests in 
light of the stringent assumptions of the various tests* 
These basic assumptions will be given prior to each distinct 
set of aiaalyses . 

PiscriiiriiTFcmt Analyses 

In asEder to utilize information on individual differences 
in predicting the potential behaviors of a student, the method 
of classification by discriminant analysis v;as selected* A 
discriminaJit analysis compares an individual pnofile wilih that 
of a group and calculates the probability of membership in 
that' group , The details of and tables for calculation of the. 
probabilities: of group membership for several variables are 
given in Appendix A, All calculations were made using the 
BMDOSm Computer program. 

The grouping variables here are primarily of two types: 
performance and efficiency/ with high/ medium/ and low levels 
d-efiiiied lEcar je^ack- The variables which comprise the pro£ile 
^or eBrflh imdivixfeial are firre Strong scores (ACH/ M-F/ OCL, 
SIN and SEL) / oim Naval of Sicer score (denoted ISIAVj / SAfT (V) / 
SAT (M) , 2igih ScJbdoI raaiSi^ (iconverted) / Whole Man Score^ 
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Quality Point Rating (QPR) for third semester ad the 

Physics Validation Score. These twelve variables and a, subset 
of them were used as descx:iptors of individuals to provide a 
relatively wdde range of loackground information for classi-- 
fication purposes* The reduced set of variables used were the 
result of a step-wise regaression analysis on final examination 
scores with SM (M) , QPRr and Physics Validation composing the 
set. 

To make tihe technique more applicable f probability function 
are presented for each group so that an instxuctor or counselor 
can obtain an individual's backgroumrd scores, substitute per- 
formance on final examination or medium level of time spent in 
a a particular week for that individtaal . . (See Appendix A for 
a detailed explanation.) The technique: (1) assures the user 
that the errors of iciisclassifyiiag each individual have been 
minimized; (2) provides a clsKck on the accuracy of the classi- 
fication when used with similar individuals; and, (3) gives a 
test^ of significance between groups over all variables. 

The variables used were assumed to be distributed as 
multivariate normal within each group and were such that the 
covariance matrix was the same f&r aH gjroups^ The validity of 
the following aamalyses rests on these assumptions. 

Tbe norm iceferenced n*ature of most: of the background 
scores gives some assurance of cBls.tr±ibution normality (at 
least isymmetryS burfc little can iae said about equality of the 
covariances- lihat tihe user of iSiese techaaiques has to do is 
remiiaia himself that y "given .the a:ssuiii5)tii3!ns are valid then 
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this is how v;e may proceed.'' How "good" it is has to be de- 
termined empirically. 



nigue of discriminant analyses see the rasi£erences on Anderson 
(1958) and Cooley and Lohnes (19 62) • 

Final Examination 

Three types of students took the ^firprrl examination, 

1. Experimental: those who were eisraa^d to the 

experimental treatments and mjsdiis.and took 
weekly pre and posttests, 

2. Control I: those v/ho took only~±iiBi: pre and 

posttests each week. 

• 3. Control II: those v;ho took nediStter pre nor 
posttests nor were e:<posed ticp-^any ex- 
perimental condition or mediae* 

Discriminant analyses v;ere condvrcrHSrTf or each of the 

types (1 and 3) above, the l = tter beizr^r^snoted the Control 



The raw final score for each stu^e— was used as the per- 
formance measure and the three levelsrxiSrmes^orTnance were: 



For a technical description and discussion of the tech- 



Group . 



Higii: 




Medium: 



those scores which wejgR— a teo 1 ut e 1 y be- 
tween X + ..5S and x — -3lS:*^ Under a nor- . 
m^l distribution this:-im*ciEades approxi- 
ina:teiy 38.% of the scores . 



c) 



Low: 



those scores -which VJKECe less than or 
equal tx> x ~ .5S. Under a^inormal dis- 
tribution this includes ajxproximately 
the lower 31% of tiie scores,. 



Weekly Posttest Perfoxmance 



For each of the V7eek.s X through G (Clasi^ 7 weeks) 
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discriminant functions were developed using the average log 
confidence score per student per week as the performance 
measure with the levels being defined as the final examination. 

Time in Media (Efficiency) 

Using the proctor's report of time (in minutes) spent in 
each student in each experimental condition as an efficiency 
measure, High Medium, and Low time groups were defined as 
above. Discriminant functions were developed which classi- 
fied each student as being high, medium, or low in time given 
an experimental condition and background variables. 

The following tables are presented for those discriminant 
analyses for which the group equality test was significant for 
a = .05 or .01. 

Tables 1, 3, 5, 7, 9, and 11 identify the performance 
grouping variable, profile v=ri = rle5, sar:?le sizes and means 
for each profile variable by grzup for the previously des~- 
cribed analyses , 

.Tables 2, 4, 6, 8, 10, and 12 display the classification 
check ..matrices giving the number of correct and incorrect 
classifications made by the discriminant functions for the 
samples used. 

For example, in Table 2, the discriminant analysis classi- 
fication procedure used correctly predicted 44 of 58 low per- 
forming control group students on the final exam given only 
their profile scores, 27 of 54 medium level students and 37 
of 54 high level students • Chi-square analyses could be con- 
ducted on each of Tables 2, 4, 6, 8, 10, and 12, but this adds 
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very little if anything to the usefulness of the procedure. 
It takes no statistical test to conclude that the classi- 
fication procedure is better than guessing. 

2 

Table 13 gives the generalized Mahalanobis D statistic 
values to test the hypothesis that the mean values are the 
same for all groups for the profile variables (See Cooley and 
Lohnes, 1962). Subject identification, grouping variable, 
number of profile variables and degrees of freedom are also 
given in Table 13. 

The classification check matrices demonstrate that the 
proportion of correct classification V7as generally above ^50 
and considerably higher than this for high and. low performance 
groups . 

The implication here is that the procedure is considerably 
better than* guessing and would be of sonie value in obtaining 
an estimate of level of success on the final examination, given 
background scores. This procedure should assist in planning 
remedials, expected tutorial tins and counseling. The object 
of the instruction would be to "beat the classification" and 
bring every student to criterion performance. 

A comparison of actual outcome with predicted classi- 
fication for each student would be a good indicator of attain- 
ment of this goal of instruction. For example, if a student 
has a high probability of being in some low performance 
groups on final examination then proper steps could be taken 
to overcome his deficiencies and bring him up to criteria on 
the final. Those students, however, who have a high 



22 



probability of being high performers could be routed through 
supplemental materials on topics which are easy for them or 
be used in a tutorial fashion with other, weaker students. 
Of necessity, the final examination would require procedural 
change if this approach were taken. Absolute criteria of 
performance would replace the relative standing criterion 
currently used. See Technical Report 4.7 page 53ff for a more 
complete discussion of this problem. 

The result, by use of step-wise regression analysis, that 
SAT (M) , QPR and Physics Validation were the best predictor 
of final examination performance is no surprise since these 
same type variables are usually best predictors of performance, 
especially QPR. 

It is apparent that these discriminant analyses do not 
identify the best predictors but only indicate what is most 
probable, given background variable of several types. Of the 
18 discriminant analyses performed only seven had D s 
suf f j-ciently large to conclude that the groups (high, medium, 
low) had significantly different population means on the 
background variables. None of the proctor time (efficiency) 
analyses was significant, but this is conceivable since one 
would not expect drastically different background character- 
ization for high, medium, or low reported time spent. 

The reader will note that the use of twelve background 
variables for classification of experimental siabjects on final 
examination performance (Table 4) . was only slightly more 
accurate than the classifications using the three best 
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predictor variables (Table 8) . For relatively rough screening 
and predictions on final examination performance it would 
appear that this reduced set of variables, representing con- 
siderable savings in testing times, and calculations would be 
quite satisfactory; we find the same variables occuring as 
best predictors in other weekly posttest analyses which will 
be discussed in detail later. 

Weekly Posttest Comparisons 

The samples of scores for the last seven weeks come from 
the experimental subjects only and the variables are total 
correct and log confidence averages on both media relevant and 
media non-relevant items. The analyses consist of step-wise 
regression each week, one-v;ay analyses cf variance with weeks 
as the treatment, randomized blc^k designs (v/ith weeks as 
blocks and media as treatments; and cne-vay analyses of vari- 
ance with media as the treatr^nr. Each will be discussed 
separately even though some analyses are closely related in 
ratior^ale and results. 

Regression Analyses 

For each week and for each of the two variables, total 
correct and average log confidence score, a linear regression 
analysis was conducted to find the best set of predictors 
(among the 12 background variables) of performance for each 
week. The mathematical model used is the one proposed by 
Draper and Smith (1966) and has assumptions of homogeneity 
of variance, linearity, and normality of the error distribu- 
tions. The analyses were conducted using the BMD0 2R computer 
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program. 

A suiTunary of the results of these analyses along with the 
multiple R values for each is given in Table 14, The analyses 
were conducted on the set of 77 experimental subjects who had 
complete data on both variables each week. The value F = 3,0 
was used for both inclusion and exclusion concerning the 12 
variables. (See Draper and Smith (1966), for explanations and 
discussions of the multiple regression technique) . 

Multiple R is used as an indicator of how good the pre- 
dictions of performance v;ould be if one used the variables re- 
sulting from the analysis. A value close to 1 is most desirable.. 

Prom Table 14 it is apparent that some of the same vari- 
ables occur again and again as best predictors of weekly per- • 
formance and include the same best predictors in general as 
were found for predicting final exam performance in the Dis- 
criminant Analysis section. The most common is QPR which 
occurs in 10 out of 14 analyses as a best predictor. It occurs 
five, times as the best single predictor. It is worthy of note 
that SAT (V) "and M-F occur several times but did not occur as 
predictors of final exam performance and that High School 
Rank, did not occur for any week and SAT (M) occurred in only 
one week and - variable only. 

If one wer J p. ;ssed to pick a best single predictor of 
performance on we^}<i^ly posttests , he could p;^obabiy do no 
better than to pick QPR and this would agree with the general 
findings of other researchers using similar performance 
criteria. This should be cautiously done, however, since 
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Table 14 shows that the percentage of total variance accounted 

2 

for using all the best predictors range from 9.6% ( . 31 ) to 
2 

33.6% (.58 ) with the best variance proportion for QPR alone 
being 21.2% (.46^) . 

These low percentage values indicate that using QPR 
alone will not yield very accurate predictions of performance 
even though under the circumstances it is the best from an 
empirical and theoretical standpoint. In other words, even 
the best predictor of perfonnance is a poor one. 

One-Way ANOVAS (V7eeks as Treatments) 

The basic assxomptions for this set of analyses is that the 
samples are indepently drawn frcn treatr?.ent population which 
are normally distributed with equal variances. Since each week 
consisted generally of the sane crcup cf subjects, there is a 
serious question of sample ir.f ep-sr.f ence* Hov7ever,,if one is 
willing to assume that the weekly sanples , representing, as 
they do, different subject matter, are, for all practical 
purposes independent, then, this relatively important assum- 
tion can at least be listed as "questionable". (See Hicks 
(1964) for details of assumptions and procedure) . Alter- 
native procedures which do offset some of these assumptions 
yielded the same general findings. 

The item^ on whi the scores are given consist of two 

types 

1. Media. Relevant items which are specifically designed, 
to measure achievement of the Terminal Objectives in the 

O ' . • - ' 

ERIC ^ 
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cotiiir^ jr which parallel media v/ere designed* Those items 
ear- ented in Technical Report 5,3: Test Item Bank, along 

wiii?"' atistical description of them. They are related to 

the ' - : al Objectives in Technical Report 5.5: Revision 
Pro^es:^ Dcumentation. The rationale for selection of the 
spec iZO's is presented in Technical Report 4.9: Design 

for -^-Blection of Strategies and Media, There was a total 

of t ^ media relevant items spread across the fourteen 

weeks the semester. 

2. *M:- non-Relevant items v/hich ar^e specifically designed 

to TVB ^e achievement of TO's on v:hxch no parallel media were 
devellP' ^ '^^^^ The instruction sources for these items were the 
textSacs: , Study Guides, lectures, a::::d laboratories. 

^>irh sets of items represent subject matter which changes 
on' ^ ^^^ly basis. Tables 15 through 20, as explained later, 
sh^: .:t^r the weeks are different, probably because some of 
tlae .■^:Jlcs are more difficult. 

^^he one-way analyses V7ere conducted on these item types 
separately as well as pooled and BMDOIV was used to conduct 
these analyses. Sample sizes in these and subsequent analyses 
vary, due to class attendance differences. . 

•Tables 15 and 16 show the results of the one-way analyses 
with weeks as treatments using scores only on the media re- 
levant Items. 

'iB^-;;::ies 17 and 18 show the results of the one-w^^y ansigrs^s 
using scores only on the media nonrelevant items while tafel^^jfe 
19 and 20 show the results for the pooled items scores. 
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The E^^nlts: shown in Ta-:)tes 15 throaigh 20 iinply fchat weeks 
are differrEsnt i^i mean score w: ^^v- i2sasured on total co;rrect and 
log confidsnice n^erage usinc ae^r^-'?!^ relevant or nonrelevant 
Items as weSIL as pooled items ■ 

Randomized Block Design Analyse 

If one csmn assiime that weeks caiL treated as xnternally 
homogene'ous ±»JLocks amd media can 3^b fiefinied as txeatimeiits / 
then the s coxes within each lEsdia group can be definieci as 
observations of a randomized bloc& design. The purpose here 
is to test for equality of the meiiia means with the re- 
striction on randomization be±ng the v/eeks . The basic idea 
is to offset tbe effect, in compsring media, that weeks are 
different. The BMD05V program was used for this analysis 
to handle the unequal sample sizes in the cells* (Hicks, 
(1964) , gives a detailed discussion of this statistical 
±echnique) . 

Table 21 shows the results o± this analysis for total 
correct over all items (both media relevant and media non- 
relevant) . Table 22 shows ±he results for log confidence 
average for the s:ame items. The assimptions are basically the 
same as for the one-way ANOlffiS in the preceding section. 

Of note is ifciie signif Icanae between. media means when 
weeks are defTrnyrd as a resSrtfiiction on randomisation. The im- 
plication is JSteL the mediae are different v;hen compared with- 
i:^ a week, knos'iill^ ^iwlt mmks are different. 

i^ne-Way Analiyses: fMeidlia as TreatmenEtS)) 

These Rtti=^l'3^gtes and thsdscr assaaaopidrCTas are latasLiicallLy 
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dite!rt5'cal to tiiase above wx' n weeks as tx©?^itinents viitiL tfhe hix- 
C2CT±rnn that tfe media now are being detSixD^od as tr:sa±EKinrs,. 
fSs- :z ^dependemiEi assumption: is slightly^ -.^aisier to niafe: iim tSsse 
a^ttSS;^ es since ^.ch media nrrcrup is compi2^s^ of dixiei:Emirh S23±b-- 

'1!!'^les 23 snd 24 show titie xesnlts z5ar media raelteT^^ 
j2iCTSr:f Tables 25 and 26 frcrisiedia nonreiievant and ZT^KE^s TV 
wmt for pocDler itaems over all weeks « 

^^nalyses ideiixical tx) those in Tffi::£i:es 27 and were 33CTn- 
€terr^£:ii within ^sci of the weeks .1 thn»:»Si^ 0.; Tables .29 Tt:hroi!agh 
43u^^nt;^7 the rsxiife of thaese analyses or both variaibl^^s on 
gcsEii^ items. 

Irt is worthy air note: tirat only 'Hzi: week L does the usasxiia 
up as siofnif icsmt wter^as when ws^efcs is taken as a ne- 

s%jrirttion on rsandiomi-zatian as in the r^jidiosdzed block Seisigm, 

^tttl^Sia shows imp as signi-iErcant. 

iriisjsai Examination 

*The final examin-ation in the coxunne ctmsisted of a fiO-item 
feffist CTontaining fonr 15-±taCTa...siibtests • The subtests we^re com-- 
posed o± constxuEted res^EHise i±ems selected 5jy two groups of 
i TT ig s tarfflggtors , andi,. multipl5er:trhoice items sell£ec±sd by ±wo groups 
JgfeiiLr uctors . 

2Kiere weare •".iiiioree basnter questions to foe ii^ted on the final 
aaQUi$33a t i on : 

!• Will expEriiUjerrbal group mean perfeinniance excess^ con- 
-^(ppoup meam -^^frfmm^^^^- on total ttet^ 

2-* VrUX ^£iter tMEgsB. t:\«3 groTips^ fiD better xxa imltiigiiLe 
c fBoA - c rg items t3hian:..:on constferoiicted response: ii^ms? 
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3. V7ill the variairce j-f the experimental group be sig- 
nificantly smaller than ±ac varii^aiice erf the control gxoTop? 
(The reasons for asking EHss-feit^te one and tliree are fiiUfc; 3as:- 
plained in TR 4,9: Des±gH frnt Selectiion of istrategies axid 
Media) • 

Table 43 presents t^s- m^i^iis and variances for the tfrrree 
major groups taking the SraaL na^^ainini^f on (experimental garoup, 
big control group, and pzsEe - p?)-?^^ coniral group) • Table 44 
presents the t-test compa!ir::^3?5^n^ and planned variance test 
comparisons on the final ei<xc.r^^^ation • 

On the total test, wixiS^. the mean of the experimental 
group was higiier than tha± ':the control group, it was isot 
significantly so. On the :tXD"t^ test, the experimental giznaup 
answered constructed re sp osaoa^ jxteius about as well as it am- 
swered multiple choice iten^.^ This condition did not obtain 
for the big control group, ^^ich answered multiple choxtce 
items significantly better .^p<^01)-. 

.The variance test befc&jre^n tthe big control group and the 
experimental group indicated: -tstia±. Ifhe experimental group 
variance was significantly snallesr (p<,01) than was the irari- 
ance of the control group* 

Audiovisual - Non-audiovisTB©:l Q3iSip:aris:ons 

For these comparisons tlshinaes ex^sci^ental groups,, AmSio- 
visual. Talking Book, and lEIEIlEistrated: Book were pooled to ^orm 
the "AV groups". The other foasn: experimental groups^ StucLy 
Guide, Lecture Demons trationt^, lectijme:, and' Student O^vtlosiB were 
• pooled to form the ''non-ESH ^rxmgs^y. For each week t-tests 
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were ooTHincte;d ±o t3e^t_ the eguaJLit^r ^^r: i^aixB for AV — xnfi non-AV 
pooled -grxiups on iiEng:. .^ xelevanf: itsns^ 'axrly • The sevB:.i weekly 
results are shiDwn ixn TabXes 45 annc^i 45 fnzr both total crzirreoi: 
and mean log rspnfn /cfence scores, '?f?he - ' ' -^ i^ ^^ -^ 'T scores, tfesarrr is 
total media r^JBV'smi it^ms for aJ.1 grnups v/ere te3St:sd 
against total unedira relevant items fxrr i non-AV grouo^. 
Since, in this case; ga'ch siibject -was in all of the com£Ltions ... 
the resulting t-tsst iaas each siii3Dject hiis own contrciX. The 
results of this aaiaiysis are shown li^ Ta&nS*B 46, for Hie^ir. log 
confidence scoxres omly. The surrerziifcdty (0:f the non~iW ^xonps 
iseegn clear. 

Two of the criteria for seledSnra. t?erTgTtinal ob3^rrf:i^es ±rDr 
-which parrallel media sequences^vouJSSae- dssyeloped wens irhe 
difficulty of the concept, and, tfhe ^eLstisre depenfienrry of ttee 
concept on visuali:2eS motior. for e 'f active learning.. A sub— 
set of BBsdia relesr^aiart items v^hs s^jscifically desigcieacL. to coe? — 
pare motdrion'-depeinleait versus notitoini-iiafle^CTadent 1fO*s. Anotsasr 
subset of test iiaeras was designed to cass^^smB diSSnm^lr: .and 
non-difficult medisa^ relevant TO's,^, 

In l>oth cases; ,, motion d^enSsaicy amd tSilf f icnflity , .At -was 
expected iiiiat: tfce ^amcslTel iDEdia g^sms^sE mmld has^^ saj^srior 
perf ormaiace: -mm tiioBe ^ecif ic tes*: :i±:eMs». In n^^iMiisBr^case was. 
this preaiction sm&iKteatiated by tte daita Obath tr'^ <: 

CorrelatioQEal Desaar^^tion of Variab3cgi 

A cor^celLatiorKaHL description ©f el^gttet^aagft vari^jfe^ me^ 

1 

""See T.iL. 4.9:; Design for Select^^OT Strrategies aaad M^Ss 
Sdx E3^bor3±ii!on., p. 25. 
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in the izirr ^^edii2ig analyssis is given in Table 47. 

Tfe 'X-^Tiables are SAT (M) , QPR, Physics ValidatinHi, finalL 
e^aitii naTt^ : - score and tDtal correct: a^id log confideaice average 
for each '^^"^z tthe last se^n.: weeks. Only those subjects which 
had cozmp2a.i:t£ da±a in all 3-ai7en weeks ("N =^ 77) were 4ise.d. 

Evesx. rthrou^h tihiLs c-wcription is not inferential it ^oes 
point out tt22xs relal3:ve-_.y -higii correlation beiiwesn tihe txDtsl 
correct amt the log confidence each week. Tiie innpiicaticnii is 
that both scoring sy:s tenis raaik subjects in a veary similajr 
maniaer on ^eacf ormance . 

The csnarrelatioiaal daiia: ^rssented in Table ^ isnd 47a were 
colli ec tedi JEor two pxincipaQ. rreanons : to investigate the re— 
lationah!%3 feetween backgrxicmd variable scores and perio3CTtanice 
in tke ioteo::^, an£i<r to otctsija mn estimate of the imter^-rie- 
iati-OiKSliipSi 3Detwe^:A perforirriaaiiE^e measures, testing piroc:ediiE3!cs.s / 
anS jr^atdiiag: ^sGTOced Tafeiife -4'7a presents a diff e^E^ant mtix: -of 

bacfegsnasmoi irariablfiES (th^an-^ifiaaes Table 47) and, a.dcfe. -two 
aiSiialiiDinsIL pesr f ormmic e vax±s)ies , total imedia xeliS?^Htt isjsfc: 
HC2are, lanafl tcital pos tt e st: ,s;cox.e . Of particular xntexest tIhe 
THpfrgrttiaiE^'^y low corx^latioa, .25 bet^^een experimental grows 
•f^^iag^l exawndlnati on perf onaaanrre and total posttest perf ornMrnce - 
Study fflriiagp ^alysis 

Tafclffis: 52 ^d 5E3 presserct tSie sumiua-riJied data Sor tllie 
Study to^l^raes. SM^ tfl^ Study Guide was tihe psriiaEiipaiii 

oirganizainig feattiia^e tcrf the .a«w«fcse , a s^iecial anialysis wais 
m^uase of sfenaeut performanosv-on the Study (Suicfes. Faisrst, 



it was desEin^ea to knc3\v wtnether stmur^fent learning occured as 
a resui-t a?f usimg the Sturdy Guides. 

Pfc^vsiicrs vg:,L2^stionLS wer^ pra^nteer to -the student and he 
was asked ±ci aainaicate his artsv/er: z>j nmrking a wet-to-rewal 
amsver siEet:.. xf his fixst choice was conrect, he was 
directoa t33 life next prroblem. Xf he was wrong, he was 
directera tcD a page witli a reniedial.. ffi.e was then asfced to 
answer the CTas'sd-iom cnxrectl^,. ZLi he was right, he was sent 
to the nisxt ^nrcttblem; if wron^, he was sent to another 
remediai.. Be could miss eacia g:uiestiOin a jnaximum of three 
times . 

JES pars carance iwBfe operating, tfoe distribution shoald 
be "follows 3. 

25* answer correctJy om the Socst trial 

?3 iL/3% vcS. -those reMaisamg isKomld a.nswer comectiy 
on Irm^ s^erond trial 

30% of tiMts^e remstiming OTaiiiliS stnswer correctly 
.on the tiE&ird trial 

iOOH: w^iulg have answered cnDrrsectly after the 
« ::f ourtfeL tnial 

Chi-Sqij^e^S 'with these theoT^ioaai frequencies were 
caLcaiLated for eaKrh of the vbliames A through O. The results 
of tfttese Chi Sg.MX%& ar^- preset ta4 a¥) T^le 53. As can be 
m^VL^ each 43f «^ 1iE>l^sl«KSS had (Slti iSegaares sufficiently high 
bo jre^ect tte :cEi™ce mstxibatiMnL .irypatiresiLs ait well beyomd the 
.1)1 level C3gf contSsdence. Learaaibng was ctesd^ demanstrat^d. 

When- wlianmer K was compareti to tfee rBEiaeiiiiing ti;6l'umes 
pooiliaa, :d:n am attempt to see if iJne diiEfereBEfc format of 
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volxome N prodiaced beltter results than the fomat for the re- 
maining wolmnes,, the Cfiii-S:guare value was 8.34, which fails 
to reach the established adpha level for a one- tail test. 
Analysis of Table 52 reveals that volume N naight be slightly 
inferior^ in that a.11 -TiriDluEnes pooled had a .6 proportion 
passing for the three puncfo Gxolnaim, while volume N had only a 
. 5 proportion passing^ 

Learniiig Category^ Confidence ^ and Difficulty Ratings 

When tlae project began, both BIooiev and Gagne concept- 

ualiz^ations of hiexajccaaical orders of learning were analyzed 

to attemprf: to fit InstacEEction appropriately to the categories. 

Fimkel (1969) coneliadlea that the Imstnaction in the physics 

couirse was at &ie. Mghest level im @aic5i of the systems men- 

iconed. It was, therefore^ deciited itiiat levels of problem 

aiff iculty^ as rafed lay physicistev womld be s\ibstituted for 

tihe hierartfeies of Hear mi ng. 

The classification of questdLoBS was done according to 

following schenne: 

Ii^ansaing Category (See exajfttg^les im Appendix B) 

0: aacogjiiifcicKn caif fDoriia ctiples and recall of 
fmcts; saiosstitiLEitiGEai ±ni formulas 

1: Substitotion into ai^ solving of single 
step proMLjems 

2: Solvfiicg of lUiUltiple aiLep problems 

3z 35ie *©the3:"" category into which questions 
were- classified whesa t2iey did not fit the 

otiieir "Steoee 'Categcmiies.. 

.■0 

lable 50 prfe&fflauts the interooirarelations, means, standard 
deviations, and fox all testt it^ rating data and 
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performance (mean proportion correct) . Coliamn 1 is mean 
proportion correct for the pooled experimental groups. 
Colxunns 2 and 3 are the faculty's item difficulty ratings in 
math and physics respectively. Column 4 is mean student 
recorded confidence, and column 5 is the students' mean 
difficulty rating. Column 6 is Learning Category, as 
described above. 

Correlations between ratings and performance should, 
typically, be negative. The rating scales used make rated 
difficulty increase on five-point scales with 5 being most 
difficult. It was expected that a higher correlation betv/een 
learning category and perf orriance would have been found. 
While both students and faculty agree on the apparent 
difficulty of the test items, and, the relationships between 
learning category and the ratings are consistent and 
significant, (for 95 df, r for the .05 level* is .17 and for 
the .01 level is .24), learning category and performance are 
not significantly related. 

Table 51 presents the . proportion passing test items of 
each learning category for the experimental groups and the 
pre-post control group. Generally, the proportion passing 
decreases as the nominal learning category increases, if one 
excludes category 3, the "other" category. The critical test 
is in the comparison of the pooled experimental groups and 
the pre-post control on media-relevant items. The per- 
formances are clearly not significantly different, and, even 
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if they were, it is hard to imagine how such small differences 
would be important for decision making* 

Both the pooled experimental and the pre-post control 
groups had higher proportions passing on media-relevant items 
than on non-relevant items, which might indicate that the . 
media- relevant items were easier. Since the pre^post 
control did as well as the pooled experimental groups, and 
did not have the benefit of the media, an item difficulty 
differential appears reasonable, if it v^ere not for the fact 
that the mean log confidence scores for both kinds of items 
shows a difference in the opposite direction. C<^relational 
data (Table 47) indicate a high relationship between proportion 
correct and mean log confidence. 



^Vnalysis of Preference Data 

Preference data v/ere collected for two reasons: To find 
out which experimental conditions students preferred in general 
and to obtain data on which revision decisions could be made. 

A thirteen item checklist was developed for each media 
group. Each item was rated by the students on a five-point 
scale from "highly favorable" to "highly unfavorable". Ten of 
the thirteen items provided the student the chance to give 
a favorable or unfavorable reaction to the specific experi- 
mental condition and three items were designed specifically 
to check on unique features of each of the conditions. The 
three specific items were intended for revision purposes of 
the media, and were not classifiable as favorable or 
unfavorable. 

The checklists were administered by the proctors in the 
various teaching areas on Monday of each of the last seven 
weeks of the semester. Each week, the students rated the 
experimental condition of the previous week. 

Two separate analyses of the data were performed. First, 
ratings of all students, were tallied for each of the 
experimental conditions across all seven weeks . These 
ratings were then combined across questionnaire items into 
one overall total for each of the experimental conditions. 
Table 4 8 presents the percentage of students, combined 
across weeks and items, in each of the experimental condi- 
tions responding to each choice on the five-point scale. 
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The choices have been ordered from favorable to unfavorable, 
even though the scales were alternately reversed on the 
actual questionnaire to prevent position responding. 

Two rank ordering© were made of this data. First, 
columns 1 and 2 were combined into a single "favorable" pro- 
portion. Then, responses 4 and 5 were combined into a 
single "unfavorable" proportion. The result is a ranking of 
the experimental conditions in terms of "most favorable" to 
"least favorable" and a second ranking in terms of a "least 
unfavorable" to a "most unfavorable" order. Neutral responses 
were also ranked with the lovrest proportion being assigned 

a rank of "1". 

Tables 48 and 49 present the proportions of students 
responding by combined categories, the ranking of these pro- 
portions, arid the rank ordering of time data. 

The means and standard deviations v;ere calculated for 
"favorable," "unfavorable" and "neutral" responses. The 
mean ,f or favorable responses was .39 with a standard devia- 
tion of .084, unfavorable .19 and .053, and neutral .37 
and .033. 

The "Lecture" and "Student Option" conditions were 
essentially tied and were both more than one standard devia- 
tion above the mean, while the Lecture Demonstration condi- 
tion was one standard deviation below the mean. These 
descriptions apply both to the favorable and to the 
unfavorable rankings . 
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The time data show a mean time in media of 171 minutes 
per week with a standard deviation of 61 minutes. Lecture 
and Student Option are essentially tied for the top rank and 
are both one standard deviation below. mean time. 

The rank correlation between time and preference data is 
.87/ which allows rejection of the null hypothesis (rho = 0) 
at the .02 level. 

The reported times indicated the amount of time actually 
spent by the student in the assigned rooms where each of the 
experimental conditions was applied. It was not required that 
students spend an equal amount of time on each of the experi- 
mental conditions. Specifically, no time was required of 
students under the "Student Option" condition. The times 
reported under student option indicate how much time these 
students spent in the experimental rooms when they were not 
required to do so. 
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V. DISCUSSION 

The Effectiveness Report (Technical Report 5.6) describes 
the conclusions reached about overall course effectiveness. 
Generally, the conclusion w^as that the multi^-media course was 
at least as effective, and probably more effective, than the 
traditional course. The basis for the claim that the course 
is as good as the traditional course is .the virtual equality of 
control group and combined .experimental mean scores on the final 
examination (Tables 43 and 44). The basis for the claim that 
the multi-media course is more effective lies in the significantly 
smaller variance of the experimental groups, as presented in 
Table 44. 

Further^ students tend to prefer the Student Option format 
of the course, the format reconunended for final course implementa- 
tion in the Management Systems Report (Technical Report 5.4) on 
an equal basis with the lecture. Table 49 presents the rank 
orderings of the preference data. Notice particularly that the 
student option and the lecture conditions are about equal in 
their preference ratings while the "LSG" condition (Lecture 
Demonstration) is least preferred. The definition of"tradi- 
tional" instruction^must include both Lecture and Lecture 
Demonstration. It is felt that there is a good logical basis for 
pooling the ratings for the lecture demonstration and lecture 
conditions to make a more thorough analysis of the preference 
data. No other experimental* conditions can reasonably be 
pooled with the Student Option for preference purposes. 
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If one pools the Lecture and Lecture Demonstration Conditions, 
it seems clear that the Student Option condition is clearly pre- 
ferred. While it is conceptually reasonable to pool Lecture 
and Lecture/Demonstration, these combinations were not specified 
a priori and must be interpreted with considerable caution. 
.(Data from the Spring and Fall of 1970 tend to indicate that 
there is a definate preference for Student Option over all other 
conditions. These data will be reported in detail in a supple- 
ment to this report as required by the contract modification of 
January 1971. ) 

Tables 23 and 24 present the data summaries and ANOVAS 
for the media relevant posttest items. These items, 30 in 
total, were selected before the beginning of the experiment to 
permit direct comparisons to be made among the various experi- 
mental treatments. These analyses were performed both on pro- 
portion correct and mean log confidence scores (see Technical 
Report 4.7, p. 28, for a discussion of the log confidence 
scoring, system used). - The F ratios for both . log confidence and 
proportion correct were not sufficiently high to merit rejec- 
tion of the null hypothesis. No evidence was found that the 
experimental conditions used were effective in producing differ- 
ential performance. A further comparison was also designed, that 
of combining all "parallel media" groups into an audiovisual (AV) 
condition and the remaining experimental groups into a nom- 
audiovisual (non~AV) condition. Each week, t- tests were made 
comparing the AV and non~AV conditions. Table 45 presents the 
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data on all of those t-tasts conducted, with only one of 14 
showing significance at even the ,05 level of confidence. 

These same media related items were further sub-divided into 
±he media selection rationale categories described in the 
Design for Selection of Strategies ai:id Media, Technical 
.Report 4,9, p. 25. When the course objectives were written 
they were classified by the physicists as generally difficult, 
specifically difficult at the Naval Academy, and motion- 
•dependent. There was a total of 158 -terminal objectives, 27 
:Ct~i- which were supplemented by the parallel media. One concern 
wss to find out whether the students performed better on motion- 
xtependent items when .they had been instructed with videotapes. 
Further, the distinction could be made between motion-dependent 
difficult and motion-dependent non-difficult items. No 
significant differences were fcjind in the comparisons between 
motion-dependent and motion-independent items , motion-dependent 
difficult and motion-dependent non-difficult items, motion- 
independent difficult and motion-independent non-difficult items. 

While it is not surprising to find these results at the 
college level with a group of highly selected students, (Table 3 
shows the SAT Verbal and Math scores for the experimental 
groups; .the means for both math and verbal scores are above 
the 75th percentile of high school seniors who later enter 
college) , it was felt that the techniques used for selection of 
the media would be effective in adding to the performance of 
students who used them. Regardless of the way that the data 

ERLC ' 



41 



was £^2fflnd:ned , it -was not possible to discern any differential 
ef£s±is; attribut3i)Ie to media. This statement is true, 
whe n . Bf^" one maintains the integrity of a priori planned com- 
parisons, or, does repeated ad hoc analyses by pooling groups 
on numerous bases in an attempt to tease out results. 

When one examines student performance under all the 
experimental conditions empi^ed in this tryout, it is difficult 
to rihiT A :basi:S :5ag: recommenjcling the inclusion of specific 
axidLxisziisEaJ. ma J'-^r m-h 1s in the physics course. 2^parently, the 
cri l vvvri^T 5S±udeQsL 33esp:onses^, as viewed by physicists^ are 
as'sx3x±i^fed with jiroblein ^wmkiiig , as evidenced both by the 
f inail^^minaticmi :aiid the weekly pos:ttests. Audiovisual (non- 
printv) nm^a do mot -seem to imprnve ^Ste level of performance 
on these kinds of tests . 

■■^mmJ:f-s±s of the preference data adds little to the basis 
for rectj«HBHendiiE^ ai^ media. The students said they 

prefet:!P^4 the l^ectoxe and Student Option conditions and that the^ 
did n'dt^ZLxke the Lecture Demonstration condition. When 
questioned about this apparent discrepancy in preferences, a 
student explained simply: "The lecturer explains how to work 
problems similar to the ones which will be on the test. If I 
have to sit through a demonstration, then I don't learn how 
to work the problems. 

"Under the Student Option condition, I can go to class if 
I want to, or, if I don't, I can learn how to work the problems 
from the other materials." 
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These results specif icH-TTly led to the emphasis on the 
print media in the final ver^on of the course. The Study 
Guides used in the Fall of 1969 were completely redesigned and 
revised on the basis of dats collected during this tryout. 
These data and revisions a3» Tnggror ted in Technical Report 5.5: 
Revision Process Document^l3ffi& .^rarticularly the discuss±on 
on p. 17f f . 
Correlational Data 

Table 47 presents the ±ixfeeE-^az23rrelations of the back- 
ground variables and the pp^imTTSBEEgs ^szaxiables. Each weekly 
posttest used the valid conSraentss scoring: system described in 
Technical Report 4.7.- The purpiorHe the valid confidence 
scoring system is to increase reldabllity of the test by asking 
a student to indicate his sTDirjective probability of being 
correct on any test item. Colur^ns 5 through 18 show the corre- 
lation between the weekly teHtsL correct scores and the confi- 
dence modified scores: week r = .98; week J, .88; week, .98, 
and so on. 

These extremely high co r r^T^aifcxoiEsraae twe en the confidence 
modified scores and the total crrrrect scores would not seem to 
be a convincing argument for the addition of confidence modified 
scores to the simpler total correct scoring procedure. More 
clerical operations are required to convert the scores to the 
confidence scores, and, in addition, students are required to 
indicate their answer to the question as well as an estimate of 
subjective probability. Further, students typically resisted 



43 



the confidence approach until it was explained to ther. that 
they would receive no credfc unless they complied. 

Tables 21 ;and 22 presjssEEh the results of ±he raastesiEtetf 
block analysis.. Signif icasEfc media and weeks effects 5«»s» round 
on both response measures, with log coiifideu^jat achie\'-±iEg- tMe 
•established alpha level. Iftte xesaits are difS-cult ±n 
iaxterpcet , since only one uf the w^is analyzed sepais^Eily 
achie-ved even the .05 le^elL. It 5b further ^ifeicuit to see 
hawmeaSia (experimental oDMSitiEms) could ha^ been a sigjiifi- 

* 

cant ^urce ©if Tawriaticm <om toi^e^ test items, (This anaO^-sis 
was performeid usxng the total of media relevant and media, non- 
relevant items.) Tables 35 and 36 present data for week L, 
and both log- coinEidence and total ccrrect reach the .05 level. 
Since this is the only week achieving the .05 level and, in 
light of the extremely high correi^ticn between total coanEect ■ 
and mean log c:on±ide3ice (Table 47), ±t: is dif ficsult to iimm^e 
how. further attempts to intexpret tiie;^ results would be feuitful, 

Whi:je the concept of learning eataegory was not useful in 
this tryout in discriminating among students and kinds of test 
items, the contractor still views it potentially useful for 
. course development. The small number of items on which the 
appropriate classifications could be made may have had some 
bearing on the results. Further, because of the time constraint 
of the testing situation, it was not possible to control for 
the amount of . time spent on each question or to specify in 
advance how much time should be allocated. 
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Further in2s?.esti^cia:ion should occur in this area. Students 
appear to answear laadia relevant and media non-relevanrt iteirEs 
with equal facility (Table 53) , independent of learning category 
(MfiCTSi logf comfidence for media salevant was 70.93, and for 
insidia noiJi-'rele^nt was 71.90). Perhaps experience with iihe 
irfcems alcjiasB is enough to mask aBy differences that may exis±., 
jEegarditegs of the type of instzarction given. There appears to 
be no y igniflcant difference in the variances. 

Ifhy Ishere sfapHM be the disi:crepancy in mean log confidence 
scores ^^nd proportion correct s coxes is unclear. Further 
invest igatiun of this inconsistency is probably worthwhile in 
order to develop a r-eliable learning category classification 
system $or piiysics piroblems. 

It was axot the purpose of the Fall 1969 tryout to arrange 
experimental conditions which would produce statistiaalLly 
significant dif:ifefrences . Ratirer, th^e purpose was to gather 
data which vxcmldL 33e useful in reviLsing the course to imake it 
more appropadL-ate for student-paced use. Procedurally, the 
kinds of data collected must be relatively inexpensive and 
require minimum time. Successive iterations of the course aJisB 
nost lifceiy to imponuve student perf ormance if such data :canno±r::be 
UB-:ed for purposes of revision. 

Xt is one purpose of the course, to increase mean s±udent 
performance and reduce the variation in group perf ormancre . To 
that end,^. an examination of Table 44 indicates that progress was 
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naade* While the difference Ijetween means favored the experi- 
niental groups, thag dlfferenxs was not statistically sigctif icant . 

Not unexpectedly, all ^pe^rimental groups did as well on those 
Fimal Examination questions xegmring constructed response 
aanisv^Bers as they did on the nmltlgle-choice q.uestions. This 
was ^ot trnmB of tiie control groups , even though no correction 
for 5uessimsg was made . 

The experinaental conditions v?ere all apparently equally 
effective in teacliing; students the required criterion behavior. 
It should be noted, hsDwever , that the criteria were based on a 
highly limited range of responses: the working of Physics 
problems- TSiis conclusion seens warranted, regardless of whether 
one uses the noOT-rrefereiixced Pir.al ExHrrdnat±Dn , or the 
criterion-ref emsi]DEre:a total p©rstt:^t snares. In the speciaX 
case of the med^iar-rel^ated test iter-s, trhe non- audiovisual 
groups did significantly better in total performance. 

If one considers the perfomance data in light of the 
preference data, it appears that istudents are concerned with 
those experimental conditions whmch take the least time and 
which are most directly related to the content of the tests. 
For example, the li/SG condition was considerably less attractive 
to the students taaan ^ras the straight Lecture (L) group. Con- 
ceivably, wliile the demonstration may have been interesting, 
the students viewed it as having no relationship to the 
important criteria of the course, namely, the working of 
Physics problems. While the inadequacy of such criteria has 
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been discussed more fiially elsewhere CBranson, 1970) they are, 
nevertheless, vsridely used. 

The preferencB of the SO condatian may be attributable to 
the small amount erf time actually prescribed for the students 
during those weeks. That is, if the Lecturer is willing to 
show the studemt how to work profelemSy he is willing to listen. 
However, if one burdens t3ae stuiient with demonstrations, Audio- 
visual presentations , etc:* , the student seems much more willing 
to do it himself. 

Regardless off the: intergrot3ip cotnparisons , the data collected 
are quite interes±i3iag. Each TeriafeBl Objective was treated in 
a variety of ways : im the Study Ptr^ges-, textbooks , and the 
lectures., naae cri±er±g>n-ref erer^cea t^st items used to measure 
the beha^or were evaTTjated by the faculty along a number of 
dimensions: appropriateness to the TO (content validity), 
difficulty in Mathematics,* dif f^jczxlt^- ±n Physics. 

T3hes^ ratings are esxferemely vsLcsble in providing a 
.-method olpgy by which a fecuTty member can, a priori , determine 
^:he level at which his course is tau^nt. Provided that one is 
willing to accept final performance cx£' the students as an indica- 
tion of the level of soi*dsti cation oS the course, the degree 
to which, this can be specified in advance is a good indicator of 
the course "level." 

If, on the other hand, it is necessary to wait until after 
the results are in to specify the level, it appears that the 
students, not the faculty , decide what performance is acceptable^ 
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particularly, if the grades in the course are assigned on any 
"normal" curve basis. 

Our results indicate that the faculty and students are 
equally accurate in predicting student performance on the basis 
of difficulty ratings. Faculty correlations were -.43 and -.61 
between performance and difficulty, while students' difficulty 
and performance correlated -.59. 

This procedure for establishing course difficulty level 
appears imminently more desirable than a method which uses 
ad hoc student performance to determine which test items should 
be retained and discarded. The results indicated a significant 
"weeks" effect, from which we inferred that weeks were not 
equally difficult. Physicists confronted with this data claimed 
to have known all along that scr.e topics were indeed more 
difficult than others^ as is virtually always the case in 
academic subjects . 

The fact that the faculty could predict, with reasonable 
precision, the .level of. difficulty of the test items, and, thus, 
control this level of difficulty, transfers the responsibility 
of course level determination to the faculty. 

The Study Guide results were of great general interest. 
While the "Linear-Branching" programmed instruction controversy 
has been dead for many years, it appeared reasonable in this 
course to offer specific remedial frames, to which the student 
was looped, when he failed to answer correctly on the first 
attempt. Further, that more specific remedials would be more 
effective than general remedials. Volume N had "general" 
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remedials. That is, the remedial was simply a presentation of 
the correct way to work the problem. The remainder of the 
course used specific remedials. That is, each problem was 
analyzed, and the most likely, common, and probable errors were 
selected for .elaboration. The students were shown why they 
were wtong, not how to do the problem correctly. 

If a remedial is effective, it ought to reduce the probability 
of error on the subsequent attempts at the answer. Thus, if 
a student has missed the correct answer on the first trial and 
is given a remedial, he ought to have a better chance to be 
right on the second attempt than someone not receiving the 
specific remedial. f 

On the basis of this data, it was decided not to include 
specific remedials dealing v/ith student errors in subsequent 
versions of the course. Course developers v;ould concentrate on 
a more careful description of the correct way of working the 
problems. 

Finally, the very low correlation between the performance 
of students on the total of 159 criterion-referenced items and 
the 60 item norm--ref erenced Final is encouraging. Professors' 
judgment of performance on criterion-referenced items (-.43, -.61) 
is a better indicator of final score on these items than is total 
student performance on norm-referenced items (.25) used as a 
predictor. Since the posttest items had been carefully 
screened for content validity, prior to their inclusion on the 
test, and had been judged according to their expected level of 
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difficulty, it was possible to make a more accurate determina- 
tion of the actual course level of difficulty than would other- 
wise have been possible. 

Subsequent versions of the course can use the Test Item 
Bank in a pretest form and establish a baseline of student 
■performance, having available past performance on the same items 
as a comparison. It is important to note here that professor 
judgment, tempered by past experience, is the critical element 
in developing the criterion measures. Student performance alone 
is not used. Consequently, test items are not discarded when a 
large proportion of students answers them correctly. They are 
discarded when they are rated and judged inappropriate by 
the faculty. 

The results of the Fall 1969 tryout demonstrated to the 
Physics' faculty that the method of instruction was not the 
critical element in student performance, an accomplishment of 
some magnitude. Further, that students could, when provided 
with the necessary instruction and materials, achieve good 
results on their own. And finally, that if data is collected 
systematically and used to revise the course components, 
improvements can be made at each successive iteration. 
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TABLE 1 

MEAN SCORES BY PERFORMANCE GROUP 
(Control Group Subjects on Pinal Exam) 



GROUP 

I 2 3 

(Low) (Med) (High) 



SAMPLE SIZE 58 54 54 



1 


SAT (V) 


559.67 


582.76 


599.35 


2 


SAT (M) 


642.59 


660.29 


691.11 


3 


H . S . Rank 


522.05 


562.54 


582. 41 


4 


Whole Man 


57185.74 


58048 .43 


60427.98 


5 


QPR 


220.50 


257.722 


312 .22 


6 


NAV 


44.55 


45.07 


42.91 


7 


ACH 


49.98 


51,96 


50.00 


8 


M-F 


51.95 


52.20 


52.07 


9 


OCL 


54 .98 


55.59 


56.69 


10 


SIN 


44197 


44.70 


47.11 


11 


SPL 


39 .69 


40.30 


38.41 


12 


PHYS. Val. 


21.88 


28.15 


30.22 



. TABLE 2. 

DISCRIMINANT ANALYSIS CLASSIFICATION CHECK MATRIX FOR FINAL EXAM 
(Control Group Subjects on 12 Variables) 



CLASSIFIED GROUP 


ACTUAL GROUP 


1 


(Low) 


2 (Med) 


3 (High) 


SAMPLE SIZE 


1 (Low) 


. 44 




11 


3 


58 


2 (Med) 


15 




27 


12 


54 


3 (High) 


3 




14 


37 


54 
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TABLE 3 

MEAN SCORES BY PERFORMANCE GROUP 
(Experimental Group Subjects on Final Exam) 



52 



GROUP 



12 3 

(Low) (Med) (High) 



SAMPLE SIZE 42 47 40 



1 


586.83 


564.28 


607. 88 


2 


665.57 


666.28 


693.40 


3 


556.57 


573.43 


600.63 


4 


57775.85 


59709.04 


59814.03 


5 


234.76 


282.36 


316.45 


6 


45.02 


44.19 


47.60 


.7 


50.00 


46.94 


51.98 


8 


50.52 


53.79 


53.35 


9 


55.67 


52.79 


58.40 


10 


45.07 


47.64 


45. 65 


11 


38.36 


35.09 


39.40 


12 


24.71 


24 .62 


/ " 31. 45 



1 



TABLE 4 • • 

DISCRIMINANT ANALYSIS CLASSIFICATION CHECK MATRIX FOR FINAL EXAM 
(Experimental Group Subjects on. 12 Variables) 



CLASSIFIED GROUP 



ACTUAL- GROUP 1 (Low) 2 (Med) 3 (High) SAMPLE SIZE 



1 (Low) 28 9 5 42 

2 (Med) 10 .32 5 47 

3 (High) 3 1 30 40 
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TABLE 5 

MEAN SCOEES BY PERFORMANCE GROUP 
(Control Group Subjects on Final Exam) 



SAMPLE SIZE 



1 SAT (M) 

2 QPR 

3 PHYS. Val, 



1 

(Low) 



58 



642 .58 
220.50 
21.88 



GROUP 



2 

(Med) 



58 



660.29 
257.72 
28.15 



3 

(High) 



54 



691.11 
312 .22 
30.22 



TABLE 6 

DISCRIMINANT ANALYSIS CLASSIFICATION CHECK MATRIX FOR FINAL EXAM 
' I (Control Group Subjects on 3 Variables) 

CLASSIFIED GROUP 
ACTUAL GROUP 1 (Low) 2 (Med) ' 3 (High) SAMPLE SIZE 

1 (Low) 45 6 7 58 

2 (Med) 17 22 15 54 

3 (High) 4 10 40 .54 
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TABLE 7 

MEAN SCORES BY PERFORMANCE GROUP 
(Experimental Group S\abjects on Final Exam) 

GROUP 



1 2 3 

(I-ow) (Med) (High) 

SAMPLE SIZE 42 47 40 



1 SAT (M) 665.57 666.28 693.40 

2 QPR 234.76 282.36 316.45 

3 PHYS. Val. 24.71 24.62 31.45 



TABLE 8 

DISCRIMINANT ANALYSIS CLASSIFICATION CHECK MATRIX FOR FINAL EXAM 
(Experimental Group Subjects on 3 Variables) 



CLASSIFIED GROUP 


ACTUAL GROUP 


1 


(Low) 


2 (Med) 


3 (High) 


SAMPLE SIZE 


1 (Low) 


27 




. 12 


3 


42 


2 (Med) 


12 




24 


11 


47 


3 (High) 


5 




10 


25 


40 



TABLE 9 

MEAN SCORES BY PERFORMANCE GROUP 
(week J, log confidence, 12 Variables) 



GROUP 



1 2 3 

(Low) (Med) (High) 



SAMPLE SIZE 34 



1 


595.79 


586.65 


573.72 


2 


666.44 


680.00 


674.21 


3 


580.88 


577.45 


■ 575.05 


4 


58641.68 


59361. 80 


59346.07 


5 


254.50 


278.96 


293.23 


6 


43.68 


47.80 


44.16 


. 7 


48. 35 


50. 80 


48. 65 


8 


50.71 


52.49 


54.02 


9 


54.29 


57.06 


54.35 


10 


51.06 


43.55 


45. 88 


11 


35.88 


38.49 


37.44 


12 


25.82 . 


27.27 


26.91 



I TABLE 10 

DISCRIMINANT ANALYSIS CLASSIFICATION CHECK MATRIX 
(week J. log confidence, 12 Variables) 

CLASSIFIED GROUP 
ACTUAI, GROUP 1 (low) 2 (Med) . 3 (High) SAMPLE SIZE 

1 (Low) 23 ' 6 5 34 

2 (Med) 12 24 15 51 

3 (High) 6 11 26 43 



TABLE 11 
MEAN SCORES BY PERFORMANCE GROUP 
(week N, log confidence, 12 Variables) 



56 



GROUP 



12 3 
(LOW) (Med) (High) 



SAMPLE SIZE 32 42 39 



1 


596.28 


5 72.26 


593. 30 


2 


670.71 


685. 78 


648.53 


3 


578.90 


542.28 


587.69 


4 


58892.12 


58016. 88 


58403.02 


5 


252.18 


272.02 


297.07 


6 


44.93 


45. 40 


45.74 


7 


47.78 


50.47 


63.43 


8 


52.18 


52.16 


62.46 


9 


54. 46 


55. 85 


54.12 


10 


48. 75 


44.00 


51. 35 


11 


36.12 


37.04 


37.15 


12 


25.65 


27. 73 


24.69 



; TABLE 12 

DISCRIMINANT ANALYSIS CLASSIFICATION CHECK MATRIX 
(week N, log confidence, 12 Variables)^ 



CLASSIFIED GROUP 



ACTUAL GROUP 1 (Low) 2 (Med) 3 (High) SAMPLE SIZE 



1 (Low) • . 20 5 7 32 

2 (Med) 8 • 26 . 8 .42 

3 (High) 5 10 24 39 



TABLE 13 

GENERALIZED MAHALANOBIS VALUES 



SUBJECTS GROUPING 
' VARIABLE 



Control 


Pinal Exam 


Expt ' 1 


Final Exam 


Control 


Final Exam 


Expt ' 1 


Final Exam 


Week I 


Log 


Confidence 


Week J 


Log 


Confidence 


Week K 


Log 


Confidence 


Week L 


Log 


Confidence 


Week M 


Log 


Confidence 


Week N 


Log 


Con fidence 


Week 0 


Log 


Confidence 


Week I 


Proctor 


Time 


Week J 


Proctor 


Time 


Week K 

< 


Proctor 


Time 


Week L 


. Proctor 


Time 


Week M 


Proctor 


Time 


Week N 


Proctor 


Time 


Week 0 


Proctor 


Time 



NO. of 

VARIABLES 


D.F. 




12 


24 


145.95** 


12 


24 


135.31** 


3 


€ 


124.43** 


3 


6 


89.61** 


12 


24 


32.7 


12 


24 


51.8 


12 


24 


32.4 


12 


24 


35.7 


12 


24 


33.0 


12 


24 


62.2** 


12 


24 


72.6** 


12 


24 


23.9 


12 


24 


.31.4 


12 


24 


31.9 


12 


24 


23.5 


12 


24 


33.7 


12 


24 


11.5 


12 


24 


19 . 6 
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TABLE 14 

THE BEST REGRESSION PREDICTORS AND MULTIPLE 

R by Week 



VARIABLE 



Week TOTAL CORRECT LOG CONFIDENCE 



I 


SAT(V), QPR, -M-F, SIN, (.58) 


SAT(V), QPR, OCL, SIN, (.56) 


J 


QPR, (.31) 


QPR, (.37) 


K 


NAV, Whole Man, (.32) 


ACH, M-F, (.36) 


L 


Whole Man, M-F, Physics 
Validation, (.55) 


Whole Man, M-F, Physics 
Validation, (.52) 


M 


SAT(M) , QPR, (.47) 


QPR, (.45) 


N 


QPR, SPL, Physics 
Validation, (.51) 


QPR, OCL, SIN, (.52) 


0 


QPR, (.46) 


QPR, (.43) 



TABLE 15 ^ 
DATA Sm-MARY AND ANOVA 
Media Relevant Items - Total Correct 



Sample Size 140 



Mean 



Analysis of Variance 
Sum of Mean F 



$9 



weeks I J K L M N 



150 150 144 ~ 137 130 



3.3143 1.5267 .9000 .4583 1.0073 2.1462 



^""^ttltion .7206 .6625 .3010 .5000 .7225 .9970 



Squares 



DF Square Ratio 



^^WeSks 757.3208 5 151.4642 , 331.5469** 
Within 

Weeks 386.0305 845 .4568 

Total 1143.3514 850 . 



ERIC 



TABLE 16 
DATA SUMMARY AND ANOVA 
Media Relevant Items - Log Confidence 



60 



Weeks I 

Sample Size 140 

Mean 83.743 

Standard 

Deviation 17.501 



J 3J L M N 

150 150 144 137 130 

74.750 90.247 54.208 57.730 59.968 

33.402 29.202 42.160 31.683 21.229 



Between 
Weeks 

Within 
Weeks 

Total 



160733. 43SS 

784704.2328 
945437.6726 



DF 



lie an 
Square 



&45 
850 



F 

Ratio 



32146.6880 34.6168** 



4^8.6441 



t 



61 



TABLE 17 
DATA SUMMARY AND ANOVA 
Media Non-relevant Items - Total Correct 



Weeks 

Sample 
Size 

Mean 

Standard 
Devi a - 
tion 



140 150 
4.8714 4.2200 



K 



M 



N 



150 144 137 130 147 

7.7800 6.2778 4.3212 2.3385 5.1156 



1.1804 1.7869 1.1694 1.6702 1.4849 1.3210 2.1975 



ERIC 



Analysis of Variance 



Between 
Weeks 

Within 
Weeks 

Total 



Sum of 
Squares 

2471.9952 

2502.0649 
4974 .0601 



OF 



■991 
997 



Mean 
Square 

411.9992 

2.5246 



F 

Ratio 



163.1817** 



TABLE 18 
DATA SUMI>IARY AND ANOVA 
Media Non-relevant Items - Log Confidence 
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Weeks 



M 



N 



Sample 
Size 

Mean 

Standard 
Devi a - 
tion 



140 150 149 144 137 130 147 

61.949 44.410 71.272 59.381 48.306 37.012 45.859 

13.721 16.596 10.117 14.137 13.198 14.542 19.420 



Analysis of Variance 



Betv/een 
Weeks 

Within 
Weeks 

Total 



Sum of 
Squares 

121609.1115 

216964.5522 
338573.6637 



DF 



990 
996 



Mean 
Square 

20268.1852 

219.1561 



F 

Ratio 



92.4829** 



TABLE 19 
DATA SUMJ-IARY AND ANOVA 
Pooled Items - Total Correct 



Weeks 



K 



M 



N 



Sample 
Size 

Mean 

Standard 
Devia- 
tion 



140 150 150 144 137 130 147 

8.1857 5.7467 8.6800 6.7361 5.3285 4.4846 5.1156 

1.6164 2.1650 1.2549 1.7380 1.8235 1.9299 2.1975 



Analysis of Variance 



Between 
Weeks 

Within 
Weeks 

Total 



Sum of 
Squares 

2181.4164 

3365.8792 
5547.2956 



DF 



991 
997 



Mean 
Square 

363.5694 

3.3964 



F 

Ratio 



107.0440** 



TABLE 20 
DATA SUMMARY AND ANOVA 
Pooled Items - Log Confidence 
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Weeks 

.Sample 
Size 



K 



M 



N 



72 72 72 72 72 72 72 

Mean 84.931 65.056 90.056 73.472 62.708 60.931 64.167 

Standard 
Devia- 
tion 11.649 13.985 8.485.13.101 13.356 11.406 13.733 



Analysis of Variance 



Between 
Weeks 

Within 
Weeks 



Sum of 
Squares 

58521.000 



76111.000 
Total • /134632.000 



DP 



497 
503 



Mean 
Scuare 



9753.500 
153.141 



F 

Ratio 



63.690 



TABLE 21 

ANOVA - WEEKS BY MEDIA BLOCK DESIGN 
(Total Correct) 



65 



Source 

Media 

Weeks 

Error 

Total 



SS 
4 3.53 
2124.02 
3257.99 
5425.54 



DF 
6 
6 

935 
947 



MS 
7.26 
354.01 

3.48 



F Ratio 
2.09* 
102.35** 



TABLE 

ANOVA - WEEKS BY I-IEDIA BLOCK DESIGN 
(Log Confidence) 



Source 
Media 
Weeks 
Error ' 
Total 



SS 
3347.48 
117593.18 
187812.63 
308753.29 



D? 
6 
6 

997 
1009 



MS 
557.91 
19598.86 
188.19 



F Ratio 

2.96** 
104.14 
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TABLE 23 



DATA SUMMARY AND ANOVA 
Media Relevant Items - Total Correct 



Media Group A b c d e p 

Sample Size 120 124 109 129 133 116 119 

1.5000 1.5323 1.5963 1.5194 1.4812 1.6293 1.5210 

Standard 

Deviation 1.1739 i i ir>nr- ^ , 

1.1221 1.1395 1.1599 1.1781 1.1686 1.1778 



Analysis of Variance 



Sum of 



Be.tween Media 
Groups 2.0234 



Square Ratio 



.3372 .2504 



Within Media 

Groups 1135.2719 843 1.3467 

"Total 1137.2953 849 
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TABLE 24 
DATA SUMMARY AND ANOVA . 
Media Relevant Items - Log Confidence 



Media Group A B C D ' E P G 

Sample Size 120 124 109 129 133 116 119 

Mean 70.055 68.174 73.865 69.142 70.382 75.332 67.682 

Standard ' 

Deviation 34.080 34.757 31.924 33.119 3", 943 32.910 34.215 



Analysis of Variance 



Between Media 
Gi'oups 

Within Media 
Groups 

Total 



Sum of 
Squares 

5817.5554 

934645.8775 
940463.4330 



DF 



843 
849 



Mean 
. Square 

969.5926 

1108. 7140 



P 

Ratio 



.8745 
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TABLE 25 
DATA SUMMARY AND ANOVA 
Media Nonrelevant - Total Correct 



Media Group 
Sample Size 
Mean 

Standard 
Deviation 



^ C D ___E . p G 

129 150 156 137 139 

5.3643 4.9586 5.2946 4.5667 4.9487 5.1460 5.0863 

2.2923 2.2107 2.3063 2.2298 .2.2225 2.1711 2.1686 



Analysis of Variance 



Between Media 
Groups 

Within Media 
Groups 

Total 



Sum of 
Squares 

60.7822 

4911. 4468 
4972. 2289 



DF 



989 
995 



Mean 
Square 



10.1304 
4.9661 



P 

Ratio 



2 .0399 
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TABLE 26 
DATA SUMMARY AND ANOVA 
Media Nonrelevant - Log Confidence 



Media Group A B C D~ -E P G 

Sample Size 140 145 129 150 156 137 137 

Mean 54.131 52.150 53.951 49.875 52.511 53.272 53.790 
Standard 

Deviation 19.796 18.066 19.798 17.560 18.471 16.729 18.195 



Analysis of Variance 

Sum of Mean f 

Squares DF Square . Ratio 

Between Media 

Groups 2199.4086 6 366 .5631 1.0847' 

Within Media 

Groups 333553.0820 987 337.9464 

Total 335752.4907 993 



TABLE 27 
DATA SUMMARY AND ANOVA 
Pooled Items - Total Correct 



70 



Media Group A 

Sample Size 140 

Mean 6.6 500 

Standard 
Deviation 2.4668 



B C D E P G 

•145 129 150 156 137 139 

6.2690 6.6434 5.8733 6.2115 6.5255 6.3885 

2.3489 2.3479 2.3553 2.4521 2.2657 2.1952 



Analysis of Variance 



Between Media 
Group 

Within Media 
Groups 

Total 



Sum of 
Squares 

66.1386 

5467.7520 
5533.8906 



DF 



989 
995 



Mean 
Square 



11. 0231 
5. 5286 



F 

Ratio 



1.9938 



TABLE 28 
DATA SUMMARY AND ANOVA 
Pooled Items - Log Confidence 



7i 



Media Group 

Sample Size 

Mean 

Standard 
Deviation 



A B C D E P G 

145 149 132 151 154 ' 140 140 

71.800 68.852 72.265 66.517 69.675 71.364 70.407 

18.323 17.344 17.353 17.683 18,077 17.370 16.421 



TUialysis of Variance 



Between Media 
Groups 

Within Media 
Groups 



Total 



Sum of 
Squares 



3471.9590 

308633.3605 
312105.3195 



DF 



1004 
1010 



Mean 
Square 



578.6598 
307.4037 



P 

Ratio 



1. 8824 



EKLC 



TABLE 2 9 

DATA SUMMARY AND ANOVA - TOTAL CORRECT 

(Week I) 



72 



Media Group a b 

Sample Size 20 20 



C 

18 



D 

20 



E 

24 



F 
17 



G 
20 



Mean 7.9500 8.0000 8.3333 7.9500 8.4583 8.4706 8.0500 

1.9861 2.1764 1.4552 1.3945 1.4136 1.5049 1.3169 



Standard 
Deviation 



Analysis of Variance 



Between Media 
Groups 

Within Media 
Groups 

Total 



Sum of 
Squares 



6. 8125 

353. 0436 
359. 8561 



DF 



132 
138 



Mean 
Square 

1.1354 

2.6746 



P 

Ratio 



4245 



ERIC 



TABLE 30 
DATA SUMMARY AND ANOVA 
(Week I, Log Confidence) 



73 



Media Group a 
Sample Size 19 
Mean 



B 
20 



C 
19 



D 

21 



23 



F 
18 



20 



81.842 82.950 85.316 82.048 85, n7 87.944 84.000 
Standard 

Deviation 15.41, le.9le 11. 634, 11.617 12.041 11.1,5 , 



Analysis of Variance 



S:am of- 
Squares 

Between Media 
Groups 602.7800 

Within Media 
Groups 21795.3914 



Total 



22398.1714 



DF 



133 
139 



Mean 
Square 



100. 4633 
163. 8751 



P 

Ratio 



. 6130 



TABLE 31 
DATA SUMMARY AND ANOVA 
(Week J, Total Correct) 



74 



Media Group 

Sample Size 

Mean 

Standard 
Deviation 



A B C D E F G 

22 22 18 22 24 20 21 

6.4091 5.3182 5.6667 5.3182 6.1250 6.1000 5.1905 

.2.1527 2.5145 2.1144 1.7563 1.6501 2.6931 2.1822 



Analysis of Variance 



Between Media 
Groups 

Within Media 
Groups 

Total 



Sum of 
Squares 

30.2652 

666.5267 
696.7919 



DP 



142 
14 8 



Mean 
Square 

5.0442 

4.6939 



F 

Ratio 



1.0746 



ERIC 
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TABLE 32 
DATA SUMMARY AND ANOVA 
(Week J/ Log Confidence) 



Media Group A B C D E • F G 

Sample Size 25 2 6 21 2 3 23 20 22 

Mean 66,720 59.769 64.286 61.217 67.043 68.800 61.364 

Standard 

Deviation 14.772 14.836 10.011 13.003 13.313 19.264 14.578 



Analysis of Variance 

Sum of Mean F 

Squares DF Square Ratio 



Between Media 

Groups 1656.4984 6 276.0831 1.3251 

Within Media 

Groups 31S77.1016 153 208.3471 

Total 33533.6000 159 . 



ERIC 
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TABLE 33 
DATA SUMMARY AND AN OVA 
(Week K, Total Correct) 



Media Groups A b CD E F G 

Sample Size 21 23 19 23 22 21 21 

Mean 8.9524 8.8696 8.6316 8.3913 8.5455 8.6667 8.7143 

Standard 

Deviation 1.0235 1.0576 1.8016 1.1962 1.4050 1.1547 1.1464 

Analysis of Variance 

Sum of Mean p 

Square Square Ratio 

.4948 



Between Media 
Groups 4.7727 6 .7954 



Within Media 

Groups 229.8673 143 1.6075 

Total 234.6400 149 
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TABLE 34 
DATA SUMMARY AND ANOVA 
(Week K, Log Confidence) 



Media Group A B C D E P G 

Sample Size 20 23 19 23 22 21 21 

Mean 89. 950 89. 21'' 86.737 86.043 86.500 87.143 88.429 

Standard 

Deviation 8.894 9.553 14.375 10.594 12.520 11.315 10.390 



Analysis of Variance 

Sum of ' Mean F 

Squares DF Square Ratio 

Between Media 

Groups 284.2886 6 47.38X4 .3799 

Within Media 

Groups 17710.7181 142 124.7234 

Total 17995.0067 148 



ERIC 
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TABLE 35 
DATA SUMMARY AND ANOVA 
(Week L, Total Correct) 



Media Group A B • C D E F G 

Sample Size 22 16 20 23 22 20 2l 

Mean 7.4091 6.4375 7.4000 5.8261 7.0000 6.4500 6^6190 
Standard 

Deviation 1.8168 .7274 1.4654 2.1246 1.6619 1.8202 1,5951 



Analysis of Variance 

Sum of Mean f 

Squares DF Square Ratio 

Between Media 

Croups 42.7098 6 7.1183 2.5(353* 
Within Media 

Groups 389.2624 137 2.8413 

Total 431.9722 143 



Media Group 
Sample Size 
Mean 



TABLE 36 
DATA SUMMARY AND ANOVA 
(Week L, Log Confidence) 



A 
24 



B 
16 



C 
20 



D 
23 



E 

22 



P 

22 



79 



G 
21 



74.875 67.562 77.750 64.652 74.864 69.000 70.571 



Standard 

Deviation 16.791 8.189 12.806 16.859 13.625 14.703 13.204 



An.alysis of Variance 



Between Media 
Groups 

Within Media 
Groups 

Total 



Sum of 
Squares 

2784.2161 

28785. 2637 
31569.4797 



DF 



141 
147 



Mean 
Square 

464.0360 

204.1.^108 



F 

Ratio 



2.2730* 



ERIC 



•TABLE 37 
DATA SUMMARY AND ANOVA 
(Week Total Correct) 



80 



Media Group 
Sample Size 
Mean 

Standard 
Deviation 



ABC D E p G 

" " " 18 22 19 19 

5.5789 5.5000 6.3333 4.7778 4.6364. 5.3158 5.2632 

2.0088 1.5040 1.8471 1.6290 1.7606 1.6348 2.1040 



Between Media 
Groups 

Within Media 
Groups 

Total 



Analysis of Variance 



Sum of 
Squares 

36 .0959 

416.1231 
452.2190 



DF 



130 
135 



Mean 
Square 



P 

Ratio 



6.0160 1.8794 



3.2009 
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TABLE 38 
DATA SUMMARY AND ANOVA 
(Week M, Log Confidence) 



Media Group A B C D E p g 

Sample Size 19 22 18 18 22 19 19 

Mean 62.316 62.864 68.556 57.222 57.045 60.053 62.053 

Standard 

Deviation 18.679 12.422 16.343 13.113 12.890 14.524 14.393 



Analysis of Variance 

Sum of Mean p 

Squares DF Square Ratio 

Between Media 

Groups 1758.3734 6 293.0622 1.3607 

Within Media 

Groups 27999.1010 130 215.3777 

Total 29757.4757 136 



ERIC 



i 
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TABLE 39 
DATA SUMTIARY AND ANOVA 
(Week N, Total Correct) 



Media Groups A . " B C D E p g 

Sample Size 16 21'' 15 22 20 19 17 

Mean 4.5625 4.6667 4.4667 4.000Q 3.5500 4.9474 5.4118 
Standard 

Deviation 2.3372 1.9579 1.6847 1.6903 1.8771 2.0405 1.5435 



Analysis of Variance 

Sum of Mean p 

Squares PF Square Ratio 

Between Media 

Groups 42.1167 6 ' 7.0195 1.9696 

Within Media 

Groups 438.3523 123 3.5638 

Total 480.4692 129 
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TABLE 40 
DATA SUMMARY AND ANOVA 
(V7r:f'V Log Confidence) 



Media Group A B C D E F G 

Sample Size 16 21 15 22 20 19 17 

Mean 60.125 58.762 57.000 52.818 51.350 59.579 62.294 
Standard 

Deviatior, 15. 832 14.195 12 .479 12.408 11.684 14.151 11.240 

Analysis of Variance 

Sum of f'ean p 

Squares DP Square Ratio 



Between Media 

Groups 1844.0645 6 307.3441 1.7610 

Within Media 

Groups 21467.5432 123 174.5329 

Total 23311.6077 129 
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TABLE 41 
DATA SUJ-IMARY AND ANOVA 
(Week 0, Total Correct) 



Media Group A B C D E F G 

Sample Size 20 21 20 21 23 21 20 

Mean 5.0500 5.0476 5.2500 4.5238 4.9130 5.8095 5.2000 
Standard 

Deviation 2.5849 2.0119 2.6132 2.2720 2.2343 1.7498 1.9628 



Analysis of Variance 

Sum of ■ ■ Mean f 

Squares DF • Square Ratio' 

Between Media 

Groups 19.0919 6 3.1820 .6455 

Within Media 

Groups 685.1547 139 4.929i 

Total 704.2466 145 
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TABLE 4 2 
DATA SUMf'lARY AND ANOVA 
(Week 0, Log Confidence) 



Media Group A B C D E F G 

Sample Size 21 21 20 21 22 21 20 

Mean 64.095 61.714 63.800 59.762 62.409 67.190 62.500 

Standard 

Deviation 16.571 13.439 19.362 14.839 15.327 10.939 14.471 

Analysis of Variance 

( 

Sum of ; Mean F 

Squares DF Square Ratio 

Between Media 

Groups 673.7842 6 - 112.2974 .4888 

Within Media 

Groups 31936.6610 139 229.7602 

Total 32 610.4452 14 5 



s 

« 

EKLC 
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TABT,E 43 







PINAL EXAM 


SUMMARY 


DATA 




Variable and Group 


Mean 


Vari ance 




Total Exam (Experirental) 


36 ,20 


48.09 


X Q 


Total Exam (Big Control) 


35.17 


75.96 


X O 7 


Total Exam (Pre-post Control) 


35.17 


60 . 84 


/ o 


Subtest 


1 


(Experimental) 


9.51 


5.68 


X fx Q 


Subtest 


1 


(Big Control ) 


9.48 


8.12 


X o ^ 


Subtest 


1 (Pre-post) 


9.44 


7 . 13 


76 


Subtest 


2 


(Experimental) 


11.55 


3.92 


X *i O 


Subtest 


2 


(Big Control) 


9.92 


5.20 


189 


Subtesc 


2 


(Pre-post) 


11.10 


4 .33 


76 


Subtest 


3 


(Experimental) 


8.94 


5.93 


14 6 
^ 1 \j 


Subtest 


3 


(Big Control) 


Q 1 P 
. J. o 


0 cr o 


189 


Subtest 


3 


(Pre-post) 


8.48 


5.86 


76 


Subtest 


4 


(Experimental) 


5. 88 


6.76 


146 


Subtest 


4 


(Big Control) 


6.57 


8.06 


189 


Subtest 


4/ 


(Pre-post) 


5. 84 


6.25 


76 



ERIC 
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TABLE 44 

"t" - TEST AND VARIANCE TEST RESULTS lOR FINAL EXAM 



Total Examination 

Experimental vs. Big Control 

Experimental vs. Pre-post 
Subtest 1 

Experimental vs. Big Control 

Experimental vs. Pre-post 
Subtest 2 

Experimental vs • Big Control 

Experimental vs. Pre-post 
Subtest 3 

Experimental vs. Big Control 

Experimental vs. Pre-post 
Subtest 4 

Experimental vs. Big Control 

Experimental vs. Pre-post 
Total Examination 

Multiple Choice vs. Constructed Response 

Experimental 

Big Control 

Variance Test 

Big Control 
Experimental 



t-value 
.13 
.14 

.01 
.03 

6.86** 
1.57 

-.79 
1.38 

-2.28* 
.11 



1 . 52 
4.15** 



1.58** 
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TABLE 45 
AV VS. NON-AV t - TESTS 



Percent Correct 





AV Group 








Group 




Week 


b ample 
Size 


Mean 


Std. ' 
Dev. 


Sample 
Size 


Mean 


Std. 

Dev, 


"t" 
Statistic 


I 


33 


84.64 


8.7 


39 


85.18 


13.8 


-0.20 


J 


32 


69.28 


15.9 


40 


61.67 


11. 3 


2.37 


K 


27 


9^.48 


8.7 


45 


89.30 


8.5 


0.33 


L 


35 


71.26 


14.6 


37 


.75.57 


11.2 


-1.40 


M 


32 


63.75 


12.1 


40 


61.88 


14.4 


0.59 


N 


33 


62.30 


12.2 


39 


59.77 


10.7 


0„94 


0 


24 


64.13 


13.9 


48 


64.19 


13.8 


-0.02 
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TABLE 46 

AV VS. NON-AV t - TESTS 
Log Confidence 





AV 


Group 






Non-AV Group 




Week 


Sample 
Size 


Mean 


Std. 
Dev. 


Sample 
Size 


Mean 


Std. 

Dev. 


"t" 
Statistic 


I 


33 


0.82 


0.1 


39 


0.83 


0.2 


-0.0/ 


J 


32 


0 . 62 


0.2 


40 


C.54 


0.2 


1. 73 


K 


27 


0.90 


0.1 


45 


0. 89 


0.1 


0.32 


L 


35 


0.67 


0.2 


37 


0.72 


0.1 


-1.31 


M 


32 


C.56 


0.1 


40 


0.54 


0.2 


0.65 


N 


33 


0.50 


0.2 


39 


0.50 


0.1 


0.00 


0 


24 


0.52 


0.2 


48 


0.54 


0.2 


-0.37 




s 

TOTAL 














AV 


X NON-?. 


.V: t = 


-28.36, 76 


df , p < . 


005. 





Q 
H 

o 



Ocr^I^r^c^O^nc^3tn^0r-. 
ovonr^ovo^LDiHt^o 

Oinr^OO^I3CN^V£>OOCOCN 
O (N (N iH rH OJ CN CM r-' rH jH 



o 
u 



o 


iH 


iH 




00 






O if) 


CO o 


o 


CO 


VD 




in 




in 




c>j CO 


o 


c» 


O 




o 




r- 




VD vo o 


o 


CO 


m 




rH 


o 






OJ OJ OJ 


iH 



















W 
EH 



cn 
w 

< 

O 

2: 
o 

H 
EH 

p< 

H 

O 
W 

w 

Q 

I 

O 
H 

o 
u 



Q 
H 

O 
O 

H 



EH 

O 

o 
p 

H 



'^3 

H X 



ocorHinor^LOo^vDCMvor^in 
oorHmmrHrHr^-^ono^o^ 
0"^Of-HocovDinr^cNCNom 

Om^^rHrHrHrHCNCNrnmrOCN 



OCTiO-^Oa^^DinnrHOrHH^ 
OrHmmOiNVO^COJVDCNVDO^ 

ococommcNoooinr^mmroin 



OC^^O^<T>C>JCOCOinr^VDrH^CO^ 
OLncTkCNCOCMCOrHO^O^CNin^ 

or^Locoor^rHr^in ^ m <n ^ ro ^ 

OOOOrHOrHmmOOOOHiH 



I 



ovDmojcnr^oinvDm^iHCNovoo 
oincooo^rooooinvDVDinr^cMinco 
or^lnlno^lno^rn^^c^Jlno^mc^Jco 
oinf>jc>jiHf>jorH^^r-)rorom^ro 



o^r^rorHCNCNrHrHin'H^co^ror^^ 
ovDc^JCo^^^^^loo^vDo^ln^^coc^Jr^m 

O CO VDAD rHVDrHCgc>JCOVD«^Ofntnfn 



o^^lno^^f^Jr^Jo^^ooo^tnvDo^^cocor^ 

O'^rHrHVD'^rOOr^rHr^OJOOjr^iHrHCO 

ocoo^vDc^J^05;31CovDr^c^tn^^r^^^tn 

OCvJromOOOOrHrHrHOJOOOOOO 



EKLC 



06 



OrHCNoo^invDr^oo 

rHCNfn«^in\X>I>COO>rHHrHfHrHHrHHrH 



c: 
o 
o 



Eh 



CO 
W 
1-3 

§ 

M 

o 

2; 
o 

M 
E^ 

P4 

M 

o 

CO 

^ 

O 
H 
Eh 
< 
1-3 

O 
U 



Q 

M 
&^ 

:^ 
o 
u 



Eh 
U 

o 



Q 
M 
ti 

O 

u 



o 
u 



o 

M 

o 
u 



Eh 
U 



o 
u 



H 
O 

u 



o r- m 

O ro 00 

O rH CTl 

O <N ^ 



O O (N 
O U5 U5 
O 00 ro 
O cTi CNi rvi 



o CM rH m 
o in o S rH 
o o^ in in o^ 

O CO ^ CM CM 



O CM ^ CO 
O CM O ^ 

O LD ^ ^ n ro 
o Q\ ^ k:* m m 



O C O CD O rH 

o o 'T^- c 

o rH r— i o in r** 

O CM CM rH H H rH 



oo^cMOln^^Or^ 
ooo^ooln^o^^o 

OU5^CM00^(nO 
0(TlCMCMrHrHrHCM 



Olno^o^^o^o^cM^o 
0^r^o^cocMc^cMO^ 
ocoorrinooootn^ 

OrHrHrHrHCMCMOO 



Eh 














0 


in 0 




0 


0 (Ti 


0 


0 


00 0 


U 


0 




!^ 


H 





o CM CO in CO 
CO 00 o (Ti 
in ^ in ^ 

O rH rH CM CM O O 



O rH 

<Tl H rH rH rH rH rH 



CM CO ^ LO ^ 00 
rH H rH 

f 



P 

H 

O 
U 



o 
o 
o 
o 



Eh 






U 








0 


00 




0 


<o 


0 


0 


CO 


u 


0 






• 


• 


0 


fH 





00 



ERIC 



T6 



92 



TABLE 47a 



Intercorrelations of background and performation variables 



on 


those subjects 


from 


whom a 


complete 


set 


of d^td was 


V u J. J. akmf J. ^ 


N 

• 


= 77 


















1 


2 


3 


4 


5 6 


7 8 


1. 


SAT Verbal 














2. 


SAT Math 


.31 












3. 


Highschool Rank .14 


.06 










4. 


Whole Man 


.17 


.32 


.57 








5. 


Quality Point 
Ratio 


.22 


.28 


• 43 • 


53 






6. 


Final Exam 


.34 


.39 


.25 


27 


.70 




7. 


Physics Vali- 
dation 


.23 


.36 


.20 


23 


.38 .52 




8. 


Media Related 


.05 


.12 


.03 


22 


.28 .23 


.04 


9. 


Final Post- 
test 


-.03 


.08 


.09 


35 


.40 .25 


.11 .1 



NOTE: - For 70 df, the .05 level is .23, the .01 level is .30. 
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TABLE 4 8 



Percentage of total responses, by experimental condition, in each 
category of the rating scales. Column 1 is most favorable, 
Column 5 is least favorable. Data are combined across the last 
seven weeks of the semester and across all items on the question- 
naire. 



Audiovisual 

Talking Book, 

Illustrated Book 

Study Guide 

Lecture Demon- 
stration 

Lecture 

Student Option 



1 
7 
13 
10 
12 

4 

23 
11 



2 

27 
28 
23 
29 

22 
27 
36 



3 

41 
35 
40 
35 

40 
32 

3.4 



4 

21 
19 
21 
21 

26 
13 
16 



5 
4 
4 

6 
3 

7 
4 

3 



ERIC 



TABLE 49 



Rank Ordering of Preference and Time Data. Rankings on percent 
most favorable, percent least unfavorable, percent neutral. Time 
data ranked from least to most. 

Favorable Neutral Unfavorable 

Proportion Proporti.on Proportion 



AV. 




.34 


.41 


.25 


TB 




.42 


.35 


•■2 3 


IB 




.33 


.40 


.27 


SG 




.41 


.35 


.24 


LSG 




.26 


.40 


. .34 


L 




.50 


.32 


.18 


SO 




.47 


.34 


.19 




X = 


.39 


.37 


.242 




S = 


.084 


.033 


.053 



Rating 

Most 
• IF 



Rank 
Least 
U. 



Order 
Neu. 



Time Rank Order 

Least to 
Most 



L 

SO 
TB 
SG 
AV 
IB 

L/SG 



1* 

2* 

3 

4 

5 

6 

7** 



1** 

2** 

3 

4 

5 

6 

7* 



1 
2 

3.5 

3:5 
6 

4.5 
4.5 



2* 
1* 

4 
3 
6 
5 
7 



_Time 
X = 171 
S = 61 



* = +1S 
** = -IS 



* = +1S 
** = -IS 
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TABLE 50 



Intercorrelations , means, standard deviations , and sample sizes 
for test item characteristics and student performance. Sample 
size for each coefficient is shovvn in parentheses under the co- 
efficient. For 95 df, the .05 level is .17 and the .01 level is 
.24. 







Variable 


Numer 








1 


2 3 


4 


5 


6 


1. 


Performance 
(mean proportion 
correct) 


-.43 -.61 

(155) (155) 


.67 

(136) 


-.59 

(107) 


-.13 

(138 


2. 


Faculty Mathematics 
Rating 


.64 

(155) 


-.40 

(136) 


.47 

(107) 


.44 

(138 


3. 


Faculty Physics 
Rating 




- .50 ■ 
(136) 


.52 

(107) 


.24 

(138 


4. 


Student Recorded 
Confidence 






-.74 

(107) 


-.27 

(120 


5. 


Student Difficulty 
Rating 








.37 

(95) 


6. 


Learning Category 










Variable Mean 


Standard 
Deviation 


Number of 
Items 




1 
2 
3 

•4 
5 

.6 
7 


.6178 
2.2112 
3.0234 
82.9378 
3.0051 
1.0652 
7.6645 


.2459 
.8056 
.5418 
13.0649 
- .7918 
.6857 
.7668 


155 
155 
155 
136 
107 
138 
155 







I 
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TABLE 51 • 

Proportion passing for each learning category for media rele- 
vant and media non-relevant posttest items. 

Learning Category Learning Category 

Media Relevant _ Media Non-Relevant 

-0 12 3 0 12 3 

All Experimental 

Groups ,78 .74 .54 .66 .67 .62 .57 .61 

Pre-Post Control .84 .71 .53 .65 .69 .66 .60 .61 
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TABLE 52 



Comparison of Volume N with all other Volumes Pooled. Propor- 
tion of correct answers on each trial. 



ALL VOLUMES POOLED 



One 
Punch 




Three 
Punch 


Four 
Punch 


TOTAL 


37819 


16256 


7209 


4090 


65374 




16256 


7209 


4090 


27555 






729 


40^0 

1 


11299 






Proportion Correct 








• 


One Punch .5 
Two Punch .5 
Three Punch .6 

VOLU:-S N 






One 
Pianch 


Two 
Punch 


Three 
Punch 


Four 
Punch 


TOTAL 


1225 


668 


264 


197 


2354 




• / .668 


264 . 


197 


1129 






264 


197 


461 



Proportion Correct 
One Punch ; .5 
Two Punch .5 
Three Punch .5 



TABLE 53 



Chi square values for each volume of the study guide. Observed 
frequencies represent multiple punches on the study guide answer 
sheet . 



Vol. 


Observed 


Expected 


df 


x2 


A 


1540782 


1061 


2 


1452.2** 


B 


1269329 


1029 


2_ 


1233.5** 


C 


563642 


772 


2 


730.1** 


D 


825715 


845 


2 


977.2** 


E 


753781 


1060 


2 


711.1** 


F 


443253 


646 


2 


686.2** 


G 


345693 


618 


2 


559. 4** 


I 


572342 


656 


2 


872.5** 


J 


195306 


422 


2 


462. 8** 


K 


394206 


604 


2 


652.7** 


L 


300950 


503 


2 


499.1** 


M . 


258121 


576 


2 


448.1** 


N 


129850 


377 


2 


344.4** 


0 


135505 


477 


2 


282.9** 
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APPENDIX A 



Probability of Group Menbership: 

TU-l calculations of the probability of group manbership by 

discriminant analysis is acccqplished by the folla>a.ng procedure, 
th 

For the i group (i = 1,2,3, , g) the probability 

th 

that a person will belong to. the i group is given by 



^ (f . - ITBX f . ) 



Pi = 



i=l 



Vfliere 

•g = niJTiber of groups used, 
e = natural logrithm base, 

^ / K ^i - ^oi ' 

k=l 

Cj^£= coefficients in the i column of the appropriate 
function table, 

C^^= constant for the same colijrin above, 

th 

= standard s core on the k (k =^ 1, 2, 3, . . . . , v) 
.variable for the person being classified, where 
V = number of variables, and 

max f i denotes the maximum value of all the f , i = 1 , 2 
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APPENDIX B 

Description of Learning 
Category Taxonomy Used 



The taxonomy consists of four categories: 

!• Zero step questions - 

Those questions which require only the recall of 
a fact or definition, or the recognition of an 
object, fact, or definition. (see example 3-1 - 
Q2) 

2. One step questions - 

(i) Those questions which require only direct 
siabstitution into an equation (usually algebraic) 
to be solved for one unknown. (see example 3-3 - 
Q6) 

(ii) Those questions which require correlation or 
association of two or more facts or definitions 
(but not directly requiring the facts or definitions 
for problem solution). (see example 3-3 - Q3) 

(iii) Those questions v/hose ansv/ers are a direct 
logical consequence of a fact of definition. (see 
example 3-3 - Q4) 

3. Multiple step questions - 

All questions not falling into the zero- or one-step 
categories. (see example 3- Post Test - Q4) 

'4'. Other - 

Those questions judged important by physicists, but 
not fitting into the other categories. 

No distinction is made among two-, three-, or more-step problems 

for two reasons. First, the number of steps can be analyzed 

only into the intended behaviors , not the actual behaviors. 

Categorizing according to the above scheme minimizes the 

difference between intended and actual behavior. Secondly, 
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when more than a single operation (step) is required to solve 
a problem, even experts frequently disagree as to the "best- 
way to solve the problem and on what constitutes a "step." 
(Are intrinsic operations 'steps'?) Clearly, ambiguities in 
the step-counting process are much more likely to occur in 
multiple-step problems. Examples for Zero-, One-, and 
Multiple-step questions follow. 



(Example 3-1-Q2) 

"Uniform circular motion" refers to 

A any circular motion. 

B accelerating circular motion. 

C circular motion without any acceleration. 

D circular motion with constant speed. 



(Example 3-3-Q6) 



Near the surface of the moon, objects fall with an acceleration 
2 

of 1.6 meter/sec . What is the weight of an object of mass 3 kg 
at the moon's surface? 



A 4.8 nt. 

B 2.8 nt. 

C 1.8 nt. 

D 3.8 nt. 



(Example 3-3-Q3) 



A rock weighs 64 lbs. on Earth, What does It weigh In free 

space, and what is its mass In free space? (the unit "slugs" 

2 

is used as a shorthand notation for lbs. sec. /ft, a unit 
of mass. 

A weight in space 6A lbs, mass in space 0 slugs. 
B weight In space 64 lbs, mass in space 2 slugs. 
C weight in space 0 lbs, mass in space 64 lbs. 
D weight In space 0 lbs, mass In space 2 slugs. 



(Example 3-3-Q4) 

The unit "newton" is a shorthand label for the units 
2 

A kg m/sec 

B kg cm/sec 

C kg sec/m 
2 

D kg sec /m 
ft 

(Example 3-Post Test - Q4) 

A light inextensible string is passed over a light, frlctionless 
pulley. Two masses are suspended (vertically) from the ends of 
the string with mass m and the other with mass 2m. When the masses 
are rclocised they have an acceleration 

A g 

B 8/2 

C g/3 

D g/4 

E 2g/3 



